Dynamic Risk Based Testing

Risk-Based Testing (RBT) is a universal approach used to determine the right and optimal set of test cases that should be executed while assuring maximum coverage.

It is observed that the existing approaches towards RBT define the risks in a static manner, i.e. the current approach does not take into consideration the factor of re-assessment of risk on a periodic basis, influenced by real-time test results.

There has been widespread acknowledgement, within the testing landscape, towards the necessity for re-assessment of the risk factor. This is based on recent observations that highlight the point that the quality of different modules of an application under development, tend to change frequently due to several variables. Hence during the course of testing, the risk needs to be reassessed on a frequent basis, based on the actual results of the test cases being executed.

The current approaches lack efficient techniques to calculate risk dynamically, reassign the risk level, and execute the optimal set of test cases for best results.

This white paper focuses on an alternative approach to Risk-Based Testing, based on dynamic reassessment of risk and its benefits.

Industry Trend

There is a paradigm shift in terms of expectation on the value of Testing and it is a common trend that Testing/Quality Assurance is now shifting towards Business Assurance. Testing is seen to have a direct impact on the brand value, market share and topline of organizations. Operationally this leads to completion of projects within a much shorter time and also at an optimal cost.

From a service provider’s perspective, the questions customers ask are:

• The schedule for this project is very challenging and we have very less time for regression. How do we choose the right and optimal set of regression test cases objectively?
• Do we need to complete execution of all identified test cases if initial testing shows that the quality of the application is very stable?
• If I follow a Risk-Based Testing approach, is there a mechanism available to update the risk levels periodically?
• How do we decide when to conclude the testing process?

All these queries point towards performing the right amount of testing, at an optimal cost and high levels of quality, and enable a shift towards a business assurance dimension.

This trend has necessitated the need to re-engineer the focus towards optimization, which should aim at: enhancing traditional approaches like Risk-Based Testing to address varying application quality and stability and a statistical approach aligned such as to achieve cycle-time reduction. The end outcome is to assure service/business resilience.

Recommended Approach

The concept considers two key parameters for assessing the risk. The first factor is the business impact, which is based on criticality of a component or module in the application, and identified based on application-specific knowledge. The second factor is the probability of failure that is continuously reassessed based on the actual test results.

There are three identified levels for each of these factors; high, medium and low, thereby resulting in nine different riskprofiles. A test case can be associated with any of these risk profiles based on the weightage for each of these factors.

The risk appetite would vary across different applications and modules, depending upon the project phase or type of release. Definition of this varying risk appetite is handled via a rule-based approach. This approach advocates specification of the optimization levels associated to each of the nine risk profiles. This definition is considered during the selection of a test suite for the execution phase.

In addition to this, depending on the pass rate of high risk profile test cases during execution, the decision on when to conclude the test execution phase can be taken, again through a configurable rule-based approach depending on the risk appetite.

Methodology

1. Assess the Business Impact:

Assessment of business impact of a test case on the application under scrutiny can be done based on defined guidelines and the knowledge of the application landscape. This factor once defined for a particular test case is usually static.

2. Compute the Failure Probability:

Assessment of probability of failure is done by analyzing the history of test execution results. Based on the extent of actual failures observed, the failure probability can be assessed. This probability is re-assessed periodically so that the influence of actual quality, revealed via the failures, gets reflected in concurrency of the risk of a particular test case.

3. Identify the Risk Profile of the test case:

Based on the level of the above two factors, a test case can be associated with nine different risk profiles as depicted in the table below:

Of these profiles, profile 6 carries the highest risk and profile 4 carries the least risk.

4. Select optimal set of test cases:

An application release is usually associated with changes to existing test requirements. The choice of test cases for the test suite should be based on three key parameters:

•  Is the test requirement impacted?
•  If a test requirement is not impacted, is it dependent on an impacted requirement?
•  Criticality of the test requirement

The above three parameters can be associated with different levels as shown in the table below:

The extent of testing / the set of test cases, that are to be chosen for a particular requirement is dependent on a combination of levels of the above three parameters. Based on the risk appetite and the combination of the levels of the above three parameters, a set of rules shall be defined, which would be referred to for selecting the optimal set of test cases.

A sample risk table is shown below:

Depending on the risk appetite, the rules table can be included to Select or Eliminate more of lower risk profile test cases, thereby optimizing the set of test cases that would be selected for execution.

5. Dynamically Decide Test Completion:

After choosing an optimal test suite for execution, a second level of optimization can be decided at run time based on the actual quality of the build being tested. This can be defined through a set of rules that would enable dynamic decision-making on test completion. A sample set of rules is shown in the table below:

The system would include:

•  Defined roles for defining business rules and generation of an optimal set of test cases
•  Methods for automated risk profile re-assessment based on real-time test results
•  Methods for automated generation of optimized set of test cases based on defined rules
•  Methods for making a decision on test execution completion based on a defined set of rules
•  Integration with a test management system, to access the requirement details, test case details, and upload of generated set of test cases

Conclusion

This approach provides a realistic method for dynamic risk assessment. Instead of an approach of risk identification using the two components of risk i.e. business impact and probability of failure as intuitive, applying a deterministic approach to probability of failure makes the risk level concurrent and well-aligned to the quality of the application being tested at any point of time.