I recently studied and passed the test for the International Software Testing Qualification Board’s (ISTQB) Certified Test, Foundation Level (CFTL). During my career I have been a tester, managed a test group and consulted on testing processes. During my time as a tester and a test manager, I was not aware explicitly of the seven principles of testing, however I think I understood them in my gut. Unfortunately most of my managers and clients did not understand them, which meant they behaved in a way that never felt rational and always devolved into a discussion of why bugs made it into production. Whether you are involved in testing, developing, enhancing, supporting or managing IT projects an understanding of the principles of testing can and should influence your professional behavior. I have broken the seven principles into two groups.  Group one relates to why we can’t catch them all and the second is focus on where we find defects. The first group includes:

  1. Testing shows the presence of defects. Stated differently, testing proves that the defects you find exist, but does not prove that there aren’t any other defects that you did not find. Understanding that testing does not prove that software or any product is defect free means that we always need to plan and mitigate the risk that we will find a defect as the development process progresses through to a production environment.
  2. Exhaustive testing is impossible. Testing all combinations of inputs, outputs and processing conditions is not generally not possible (I was involved in a spirited argument at a testing conference that suggested in very simple cases, exhaustive testing might be possible). Even if we set aside exoteric test cases, such as the possibility of a neutrino changing active memory while your software, application or product is using it, the number of possible perpetuations for even simple changes is eye popping (consider calculating the number of possible combinations of a simple change with 15 independent inputs each having 10 possible values). If exhaustive testing is not possible, the testers and test managers must use other techniques to focus the time and effort they have on what is important and risky. Developing an understanding of potential impact and possibility of problems (risk) is needed to target testing resources.
  3. Pesticide Paradox. The value running the same type of test over and over on an application wanes over time. The metaphor of pesticide is used to draw attention to the fact that once a test finds the bugs it is designed to find (or can find – a factor of how the test is implemented) the remaining bugs will be not found by the test.  Testing must be refactored over time to continue to be effective. This is why simply automating a set of tests and then running them over and over is not an adequate risk reduction strategy.

The first three principles of testing very forcibly remind everyone involved in developing, maintaining or support IT applications (hardware or software) that zero defects is aspirational, but not realistic. That understanding belies the shocked disbelief or manic finger pointing when defects are discovered late in the development cycle or in production. They exist and will be found. Our strategy should start by first avoiding creating the defects, focus testing (the whole range of testing from reviews to dynamic testing) on areas of the application or change based on risk to the business if a defect is not found, and have a plan in place for the bugs that run the gauntlet. In the world of IT, everyone, developers, testers, operators and network engineers alike, need to work together to improve quality within real world constraints because unlike Pokémon, you are never going to catch them all.