Artificial Intelligence (AI) is impacting the way businesses are run; is transforming how people perform at work and there is a tectonic shift in almost all industries owing to disruptive capabilities being brought by AI. Some old professions are clearly getting redundant and some new professions are emerging due to AI’s interventions. However, Software Testing is one such profession where there is a widespread lacuna – whether Software Testing is a dying profession or it will survive in AI era. Lets look at how AI is going to impact Software Testing and what current and aspiring Software Testers ought to do to transform themselves to become AI Ready.
Understanding AI
Before we examine AI’s impact on testing lets look at some common definitions of AI.
Forrester defines AI as “A system, built through coding, business rules, and increasingly self-learning capabilities, that is able to supplement human cognition and activities and interacts with humans, but also understands the environment, solves human problems, and performs human tasks.”
Another definition in Forrester Glossary: “The theory and capabilities that strive to mimic human intelligence through experience and learning.”
Another interesting definition is that AI is “A field of study that gives computers the ability to learn without being explicitly programmed.”
One of the key points of AI is that you don’t need to explicitly program algorithms. Algorithms are certainly used, but they’re not designed for an explicit solution of a distinct problem. Instead, the algorithms are designed to enable machines to learn and adapt autonomously, and they use data to do so.
While we examine AI in detail here, it is imperative to understand the overall placement of AI amidst the related and equally hot concepts like ML, Deep Learning, Data Science, Big Data and Neural Networks. To put it simply, all these fields coexist and enjoy a significant interdependence like shown in Illustration 1 here:
Why AI will be essential for Testers
The transition from traditional waterfall to CI/CD/ DevOps has increased the release cadence (see Illustration 2) significantly over the years. This increased velocity has made the traditional approach towards Testing and Test Automation inadequate to keep pace with the release expectations. Conventional Test Automation procedures which heavily depend upon driving the UI cannot keep pace with too many and too frequent changes in the UI itself. In the recent World Quality Report 2019 co-published by MicroFocus, Sogeti and CapGemini 61% of the Fortune 500 respondents agreed that the biggest challenge in achieving desired level of test automation was frequent application changes with every release. One possible way one can match steps with the release cadence produced by DevOps and CI/CD is by adopting AI in Testing. Autonomous scripts, self-healing tests, automated test authoring are some approaches for doing so.
In the same World Quality Report 31-36% respondents also agreed that Understanding of AI implications on business processes, testing skills, test strategy, designing, environment setup and coding skills are lacking in staff (Illustration 3).
Core Skills Required to work with / in AI
Irrespective of the profession one is into, application of AI in any business context demands some common Core Skills around Statistics / Mathematics, Data Science and usage of specific computer languages to apply the Statistical / Mathematical concepts on the Data. Understanding of all these three core subjects is essential for applying AI in any profession. Illustration 4 shows how these skills overlap for any AI implementation.
For the context of Software Testing profession the core skills one needs to cultivate are:
Language Skills:
Python and its basic concepts like numpy, panda and packages like PyTorch, TensorFlow Python, Matplotlib / PyPlot and alike.
R and its basic concepts along with CRAN packages like infer, DataExplorer, equisse.
There are numerous packages available for R as well as Python and it all depends upon your individual needs. The idea is to cultivate the ability to pick and use the package you need for your context.
Data Science Essentials:
Data Science basic concepts like Data Science Methodology, Model Designing, Model Testing (A/B testing*) and Model Validation, Big Data basic concepts (using Spark, Hadoop, Scala, MapReduce, NoSql), Sampling of Data.
Mathematical Concepts:
As explained earlier, AI and Machine Learning implementations are done by applying manifestations of the confluence of Mathematical and Statistical concepts applied on the data. Data Science is nothing but the art of learning from historical evidences of data and predicting the future events based upon the historical data patterns. Hence, in order to get started with anything related to AI, ML, or Data Science one needs to gain good understanding of Probability and Statistics.
“Probability deals with predicting the likelyhood of the occurance of future events,while Statistics involves the analysis of the frequency of past events”
These are the common concepts of Probability one needs to understand:
- Fundamentals of Probability (event, trial, random variable, empirical and theoritical probability), Joint probability, Conditional Probability, Bayes’ Theorem.
- Understanding of Descriptive Statistics i.e the methods of summarizing and organizing the information in a data set. This includes fundamentals like Elements (also called Cases or Subjects- the entities for which information is collected). Descriptive Statistics also includes the study of Qualitative or Quantitative Variables, Discrete Variables, Continuous Variable, Population, Samples, Measurements of Center (Mean, Mode, Median, Range), Measurement of Variability (Range, Variance, Standard Deviation), Measurements of Position (Percentile, Z-Score, Quartiles). The real world AI/ML/Data Science problems often consume application of Uni-Variate or Bi-Variate Descriptive Statistical concepts in conjunction with some Visualization techniques prescribed by statistical studies like scatter diagrams, pie charts, histograms or heat maps. Most AI/ML/DataScience problems consume concepts like Correlation and Regression Analysis along with Time Series Analysis.
All above concepts are applied on the data available in order to build a Hypothesis and test the Hypothesis in order to predict the outcomes based upon the data. Data Science includes Model Designing, Model Training, Model Validation and Model Testing. Hypothesis Building & Testing as well as Model Testing are the routine tasks one needs to perform in the Data Science world. While all this was always possible mathematically, cheap computing resources available today have made it possible to perform these tasks on the data Volume, Velocity and Variety (often called 3Vs of Big Data) and hence we see the emergence of the fields of AI/ML/Data Science.
This is clearly a deviation from the traditional skills testers are asked to cultivate (for example, functional test automation tools knowledge for driving UI through scripts). Now that we understand what is AI and what all does it require to be in; lets explore how AI impacts and applies to our profession.
Pick your battle- “Testing AI” OR “AI in Testing”
When talking about AI and Testing together one needs to keep in mind the context- whether to test AI based applications OR consumption of AI in order to test any application. The approach to Test AI based application can be significantly different from consumption of AI techniques in testing a conventional application.
Testing AI:
Some of the key activities of any AI application implementation include: Model Training, Model Testing and Model Validation.
Model Training: Here the sample data (training dataset) is used to ‘fit’ the model designed for training.
Model Validation: Here the sample data (Validation dataset) is used to provide an unbiased evaluation of a model fit on the training dataset while tuning model hyperparameters.
Model Testing: Here the sample data (testing dataset) is used to provide an unbiased evaluation of the final model fit on the training dataset. This often includes A/B Testing approach wherein one tests the effectiveness and efficiency of two models-Model A and Model B are compared to select any one of them. This also includes Statistical Hypothesis Testing to infer the best model. As the data set changes progressively (read release cycles) the model and hypothesis may also require to be inferred again (read regression testing).
Due to close interconnection of these related activities it is often suggested and is prudent for a single individual or team to perform all these activities. You don’t have to be a great programmer to become an AI Test Engineer, rather you need to have basic programming skills (see core skills above) along with thorough understanding of data analytics and statistics.
Individuals having the aforementioned core skills can easily find themselves contributing in all the above activities. Testers working in such teams are often required to contribute in these activities in addition to the other conventional application testing tasks.
AI in Testing:
Application of AI concepts in Testing is entirely different topic where-in much like any other traditional testing type you need to apply core test concepts but along with some tools that not only just automate these tasks but also enable some level of autonomy. Application of AI can be at any level from Test Authoring to Execution to Reporting & Analytics. AI if applied to Testing actually enables and emphasises the relevance of testing in compressed release schedules we see today. There are various tools available today for each application area of AI in testing spanning from Self-Healing scripts (though this is not true AI as per the definition we examined earlier), auto-generated scripts, scriptless automation, using RPA bots for automating complex workflows which are not possible due to functional test automation tool limitations, applying data analytics on Test Execution Results, and Predictive Analytics. Some of the popular tools which provide such capabilities are: Mabl, Functionize, test.ai, Applitools, Testim and several more are gaining ground. Most of these test tools were created as independent test frameworks designed with AI implementation built in. You can design your own as well!
Irrespective of the case weather one is Testing AI based application or implementing AI in Testing the above core skills are essential to get started and succeed. The manifestations, mix and depth of consumption of these skills may vary from case to case but one cannot do away with them completely. Also, the real application of these skills would be done in conjunction with other conventional testing skills. So, it is safe to say that a Good tester of today can become a Great Tester in AI Era only after acquiring the essential core skills of AI viz. Language, Data Science and Statistical Concepts.
Originally published in TestAIng Newsletter.