Risk-Based Testing (RBT) is a universal approach used to determine the right and optimal set of test cases that should be executed while assuring maximum coverage.
It is observed that the existing approaches towards RBT define the risks in a static manner, i.e. the current approach does not take into consideration the factor of re-assessment of risk on a periodic basis, influenced by real-time test results.
There has been widespread acknowledgement, within the testing landscape, towards the necessity for re-assessment of the risk factor. This is based on recent observations that highlight the point that the quality of different modules of an application under development, tend to change frequently due to several variables. Hence during the course of testing, the risk needs to be reassessed on a frequent basis, based on the actual results of the test cases being executed.
The current approaches lack efficient techniques to calculate risk dynamically, reassign the risk level, and execute the optimal set of test cases for best results.
This white paper focuses on an alternative approach to Risk-Based Testing, based on dynamic reassessment of risk and its benefits.
There is a paradigm shift in terms of expectation on the value of Testing and it is a common trend that Testing/Quality Assurance is now shifting towards Business Assurance. Testing is seen to have a direct impact on the brand value, market share and topline of organizations. Operationally this leads to completion of projects within a much shorter time and also at an optimal cost.
From a service provider’s perspective, the questions customers ask are:
- The schedule for this project is very challenging and we have very less time for regression. How do we choose the right and optimal set of regression test cases objectively?
- Do we need to complete execution of all identified test cases if initial testing shows that the quality of the application is very stable?
- If I follow a Risk-Based Testing approach, is there a mechanism available to update the risk levels periodically?
- How do we decide when to conclude the testing process?
All these queries point towards performing the right amount of testing, at an optimal cost and high levels of quality, and enable a shift towards a business assurance dimension.
This trend has necessitated the need to re-engineer the focus towards optimization, which should aim at: enhancing traditional approaches like Risk-Based Testing to address varying application quality and stability and a statistical approach aligned such as to achieve cycle-time reduction. The end outcome is to assure service/business resilience.
The concept considers two key parameters for assessing the risk. The first factor is the business impact, which is based on criticality of a component or module in the application, and identified based on application-specific knowledge. The second factor is the probability of failure that is continuously reassessed based on the actual test results.
There are three identified levels for each of these factors; high, medium and low, thereby resulting in nine different riskprofiles. A test case can be associated with any of these risk profiles based on the weightage for each of these factors.
The risk appetite would vary across different applications and modules, depending upon the project phase or type of release. Definition of this varying risk appetite is handled via a rule-based approach. This approach advocates specification of the optimization levels associated to each of the nine risk profiles. This definition is considered during the selection of a test suite for the execution phase.
In addition to this, depending on the pass rate of high risk profile test cases during execution, the decision on when to conclude the test execution phase can be taken, again through a configurable rule-based approach depending on the risk appetite.
1. Assess the Business Impact:
Assessment of business impact of a test case on the application under scrutiny can be done based on defined guidelines and the knowledge of the application landscape. This factor once defined for a particular test case is usually static.
2. Compute the Failure Probability:
Assessment of probability of failure is done by analyzing the history of test execution results. Based on the extent of actual failures observed, the failure probability can be assessed. This probability is re-assessed periodically so that the influence of actual quality, revealed via the failures, gets reflected in concurrency of the risk of a particular test case.
3. Identify the Risk Profile of the test case:
Based on the level of the above two factors, a test case can be associated with nine different risk profiles as depicted in the table below: