Across industry verticals, enterprises are moving towards test automation. It has been observed that there are a lot of investments into test automation but not every initiative has been able to provide the desired benefits. How many times have we heard the project sponsor say – “I have invested into automation, but I don’t see the benefits and automation has not been able to add significant value.” What moved the benefits away? The answer lies on the Automation Checkerboard.
Test automation has characteristics similar to the news we read in a newspaper or watch on television. It keeps changing every day, and it needs to be refreshed regularly, lest it loses its intrinsic value. Usually, test automation is an afterthought largely driven due to schedule slippages in upstream development. The typical approach to automation is an ROI computation that compares the investments against the effort savings brought in by automation. Developing an automated test upstream is cheaper than developing the same script much later in the development lifecycle. Due to this dynamism, the traditional ROI model will not predict the benefits correctly.
Every enterprise believes that it can derive significant benefits from automation. However, there is no structured model or approach, which defines where to start and when to end the investments into automation. The complexity of the problem is aggravated due to the fact that test automation activity is highly dynamic in nature. Any task that gets automated will always be affected due to a business change, process change or technology change. Hence a simple ROI model will always fail, and will be inadequate to drive the strategy for automation.
Understanding the problems with test automation
Let us look deeper into the problems of test automation at an enterprise level. Here is the story of a test team – Team GodSea which is performing the testing of a suite of applications in a leading investment bank. The classical problems with test automation are brought out.
GodSea - the test team has been writing regression tests and executing them manually across many releases. The regression pack has grown, and the test assets have changed many hands. It has reached a stage where the regression pack is large, and cannot be executed entirely for the next release as manual manpower is not sufficient. The team selects a subset of tests for regression based on the risk and priorities. The regression pack is not fully executed, and the test coverage is limited. The natural fallout is that some bugs slip into production.
GodSea test team figures that the best way to solve the problem is to automate the regression test pack. Performing an automation assessment yields the following results
- There are 12 releases in a year. Since the automated tests are repeatable, the ROI for automation is perceived to be good
- Based on the technology stack, a popular choice of automation tool is determined
- Availability of skillset is also considered during the tool selection proces
A new test automation team MountainGod is setup to address the issues in testing. Their objective is to bring in cost benefits along with increased quality. Once the automation team starts, it faces the following challenges:-
- Understanding the test scenarios has a dependence on the SME and manual test team
- The automation team consists of technology specialists and they are unable to distinguish between what tests are valid and which ones are outdated
- While the automation development is in progress, the test scenarios are continuously changing. Hence an automated test fails in the test environment as new changes in the application have come in
- Unavailability of the test environment and test data pose additional challenges to the automation team towards automated tests development
- Legacy technologies and test cases requiring a lot of manual intervention make automation development more difficult and more expensive
- The development team is building new features in an Agile development process. The automation team is expected to keep pace and finish up the automation of the regression pack, much faster
A periodic review of the MountainGod team reveals that the automation is far behind expectations and the confidence of the application teams on the overall testing is low. Additional efforts have gone in to make the automation work, making the automation more expensive and the ROI weaker. The automation scripts that have been built so far are being A new test automation team MountainGod is setup to address the issues in testing. Their objective is to bring in cost benefits along with increased quality. Once the automation team starts, it faces the following challenges:- subject to continuous maintenance. The GodSea team is scampering as testing schedules are crunched and they have to finish up in the reduced timeframe. The benefits from automation are being questioned.
This is the classical story with every enterprise on an automation journey.
The Solution- Automation Checkerboard
The benefits derived from automation for an enterprise needs to be looked at in two dimensions. The first dimension is the ‘Quantum of Automation’ and this drives the classical ROI model. The quantum of automation achieved resulting in effort savings compared against the absolute manual effort determines the cost benefit achieved through automation.
The second dimension often neglected is the ‘Effectiveness of Automation’. In the purest sense, effectiveness of automation is a misnomer, as it tends to define the effectiveness of the tests itself than the automation per se. However, automation without this relationship to the defined tests is quite meaningless and hence it is imperative to keep this dimension into consideration.
Automation has evolved from being an efficiency lever into a two dimensional plane where the effectiveness of automation also needs to be measured. This can be best visualized as a simple two dimensional checkerboard.
The starting point on the grid is the current state of automation in the project. If a project has moved from 20 to 80% automation achievement (X axis), it is highly possible that due to maintenance considerations, the effectiveness can be near zero (Y axis). Using the two dimensional automation checkerboard Automation Checkerboard gives a clear answer to the question – Where and how the benefits move.
The Automation Checkerboard is a useful tool to define the automation strategy and roadmap for an enterprise – the plane on the x axis defines the quantum and the y axis defines the effectiveness. The right moves by an enterprise will deliver on both planes – efficiency and effectiveness, and hence deliver maximum value from automation. To be able to effectively use the Automation Checkerboard, it is important to understand the positioning and the moves towards realizing the benefits. The aim would be to move from the current position to the top right corner to derive the maximum benefits.