Test Case Prioritization Using Code Coverage Analysis

What is Test Case Prioritization?

Test Case Prioritization is a method in which the execution order of test cases is scheduled to maximize software testing efficiency.

Consider a common dilemma in the software development lifecycle: testing must be conducted, but there is not enough time to run the full regression suite. Through a prioritization approach, one can schedule test executions such that only a subset of tests covering the most source code is run. This method reduces the time required to test the software, while maximizing testing efficiency.

But how can we determine the correct subset of tests? And, by extension, how can we determine the source coverage provided by our test suite? The answer: using a code coverage tool.

Code Coverage Analysis

froglogic's Squish Coco is a multi-language code coverage analysis toolchain. Using automatic source instrumentation, Coco can determine the source coverage given by manual, unit, GUI or integration test executions. It offers a feature specifically for Test Case Prioritization. Coco's CoverageBrowser program will calculate an execution order for tests with which high code coverage can be reached quickly using a small number of tests. Coco will provide an execution order in which the test with the highest coverage is listed first. The second test is that one which makes the additional code coverage as high as possible, and so on. All of this is calculated automatically, with a push of a button.

We'll look into Coco's method of Test Case Prioritization using a simple C++ program which acts as a parser (or, calculator) for basic expressions. This example is included in all Coco packages.

A Step-by-Step Example

The parser example includes a set of unit tests. In order to make use of Coco's Test Case Prioritization, you must first instrument the application and execute all unit tests once. The documentation includes a step-by-step guide on doing just that.

Once the parser program is compiled with instrumentation and the tests are run, we'll view the coverage results in the CoverageBrowser. Two files are generated after executing the tests: unittests.csmes and unittests.csexe. The former contains information needed for the coverage measurement, while the latter contains the results of code execution.

Load these into Coco's CoverageBrowser:

<code>$ coveragebrowser -m unittests.csmes -e unittests.csexe</code>

To calculate an optimized test execution order, follow these steps:

1. Select the executions in the Execution window. (In this example, one test failed -- you can deselect this unit test.)
2. From the top menu bar, choose View > Optimized Execution Order...
3. In the window that appears, click Compute.

The window displays the prioritization. It shows the aggregated coverage level and aggregated time spent executing these tests. Note: for each new line in the results, it displays the cumulated coverage. For example, the unit test testVar provides 29.208% code coverage with a compute time of 0.000681 seconds. Line 2, testSyntax, lists the additive coverage of both testSyntax and the preceding test, testVar, at 32.921%.

We selected 13 of the 14 total unit tests to prioritize. However, you'll notice only 11 unit tests are displayed in the Optimized Execution Order window. The optimization only focuses on coverage. That is, if two tests cover the same line of code, Coco will recognize them as redundant and will ignore one of them. This makes sense for e.g., a smoke test, but is not recommendedfor a full functional test run.

Optimized Execution Order

Now, in future rounds of testing where time is a limiting constraint, you can select only those tests which provide the highest coverage in the shortest amount of time. This proves useful also if, for example, your testing method includes many manual tests.

Recommendations for Future Reading

Test Impact Analysis, another optimization method to improve testing efficiency, is used to determine which tests execute a specific code change. This analysis is useful when, for example, you've committed a last-minute patch to the source code, but there is not enough time to run the full test suite. You'd still want to conduct some risk assessment. With Test Impact Analysis,  you can specifically display the code coverage for the changed lines of code, and find the tests in a large suite that cover them. We've written an article on this topic, which you can find here.

Summary

Test Case Prioritization can save a lot of time and effort, especially in development scenarios where time is a limiting constraint. With a code coverage tool like Squish Coco, you can determine the optimized execution order with a click of a button.