What is Mutation Testing?

Mutation Testing is a powerful method for finding errors in software programs. It was introduced as a way of measuring the accuracy of test suites. Generally, there is no easy way to tell if the test suite completely tests the program or not. If the program passes the test suite, one may say that program works correctly on all the cases that are included in the test suite. The more cases a test suite contains, the higher the probability that the program will work correctly in the real world. However, there is no method to measure how accurate the test suite is and the probability that the program will work correctly.

In mutation testing, one is in some sense trying to solve this problem by inverting the scenario. The thinking goes as follows:

Let’s assume that we have a perfect test suite, one that covers all possible cases. Let’s also assume that we have a perfect program that passes this test suite. If we change the code of the program (this process is called mutating) and we run the mutated program (mutant) against the test suite, we will have two possible scenarios:

• The results of the program were affected by the code change and the test suite detects it. We assumed that the test suite is perfect, which means that it must detect the change. If this happens, the mutant is called a killed mutant.
• The results of the program are not changed and the test suite does not detect the mutation. The mutant is called an equivalent mutant.

If we take the ratio of killed mutants to all the mutants that were created, we get a number that is smaller than 1; this number measures how sensitive the program is to the code changes.

we got another scenario when in the case of not having perfect program and perfect test case suite, the below scenario will be raised.

• The results of the program are different, but the test suite does not detect it because it does not have the right test case.

If we take the ratio of all the killed mutants to all the mutants generated, we get a number smaller than 1 that also contains information about accuracy of the test suite.

Benefits of Mutation Testing:

• Automatic mutation testing brings a new level of error-detection to the software developer.
• Tools that automate mutation testing are able to uncover ambiguities in code previously thought impossible to detect automatically.
• By using tools that incorporate mutation testing into state-of-the-art error-detection technology, developers are able to flush out more faults than with any other technology.
• Software developers and testers using tools that incorporate this approach to mutation testing will benefit enormously, as such tools automatically uncover more bugs than any other technology.
• The customer also benefits from mutation testing, as the program a user receives is less buggy and more reliable. This increased confidence will in turn benefit your company where it matters most- the bottom line.

Related Topics:

• static testing
• black box testing
• Types of Performance Testing
• Dynamic Testing
• white box testing