[Read] Seven Principles of Software Testing

Seven Principles of Software Testing

"Testing is about producing failures."

Principle 1: Definition

"To test a program is to try to make it fail."

1. paradox: very small pins, very large doll ⇨ set realistic expectations
2. Defination of testing

1. the IEEE standard terminology:
   • failure: An unsatisfactory program execution
     • pointing a "fault" in the program
     • the result of a "mistake" in the programmer's thinking
   • bug: any of these phenomena
     • informal term
2. A successful test is only relevant to quality assessment if it previously failed.
3. A failed test gives us evidence of non-quality. If the test previously passed, it indicates regression and points to possible quality problems in the program and the development process.
4. The single goal of the testing process is to uncover faults by triggering failures.
5. Testing does not deal with correcting faults, only finding them.

Principle 2: Tests versus specs

"Tests are no substitute for specifications."

Tests are instances, they miss the abstraction that only a specification can provide.

"extreme case", "generation"

Principle 3: Regression Testing

"Any failed execution must yield a test case, to remain a permanent part of the project's test suite."

regression: the propensity of previously corrected faults to resuscitate.

regression testing: Checking that what has been corrected still works.

Principle 4: Applying oracles

"Determining success or failure of tests must be an automatic process."

test oracles: A test is only useful if you can unambiguously determine whether it passed. The criterion is called a test oracle.

The form of oracles is specified separately. A test oracle is often in the form of executable assertions such as in the JUnit testing framework.

Principle 4 (variant): Contracts as oracles

"Oracles should be part of the program text, like contracts. Determining test success or failure should be an automatic process consisting of monitoring contract satisfaction during execution."

Principle 5: Manual and Automatic Test Cases

"An effective testing process must include both manually and automatically produced test cases."

Principle 6: Empirical assessment of testing strategies

"Evaluate any testing strategy, however attractive in principle, through objective assessment using explicit criteria in a reproducible testing process."

Testing research is vulnerable to a risky thought process.

Testing is tricky; not all clever ideas prove helpful when submitted to objective evaluation.

Principle 7: Assessment Criteria

"A testing strategy's most important property is the number **of faults it uncovers as a function of **time."

There is little actual evidence that higher coverage has any bearing on quality.

\(f_c(t)\) —— fault count against time

• using a software base with known faults ⇨ how many faults it finds in a given time
• feed \(f_c(t)\) into a reliability model to estimate how many faults remain ⇨ addressing the time of stopping testing