A basic testing strategy overview in 5 minutes

Your testing strategy will form the basis of your requirements validation and verification, and is essential to providing both developer and business confidence in the system produced throughout it's lifecycle. You'd be hard pressed to find an individual or company within the software industry that does not buy into this notion, however it soon becomes apparent that different individuals and companies buy into different testing strategies and to different degrees. In this article I will lay out the pros and cons of different levels and types of testing and provide a basic strategy that should provide fairly strong developer and business confidence in the system created.

The testing triangle

No article on testing would be complete without mentioning the testing triangle. Across the web you will find many variations of this diagram, however the essence remains the same - when it comes to functional testing you should have many more low level "unit" tests than any other kind of test, a good number of tests around the points of integration within your code, while tests at a high level that exercise the system as a whole should be kept limited.

This is because high level tests are slow to write, brittle and therefore difficult to maintain. They're slow to run, and often non-deterministic and error prone, undermining confidence in them. On the other hand, unit tests are quick to write and far less brittle due to their isolated nature. They're quick to run and deterministic. That's awesome. However, try taking your unit test coverage to "the business" before your next go live and see if this alone provides sufficient confidence.

There is no holy grail in testing. A comprehensive strategy will provide the greatest value and highest overall confidence.

Unit testing

I often think of unit tests as developer tests. These should be factored into estimations, and provide a way for developers to verify the correctness of their code, providing confidence in their work.

Without these, developers often rely on testing the different paths through a unit of code by manually exercising a UI or an API, which is slow and will most likely fail to isolate the code under test. What's worse, they will often forget to test all of the paths through their code after each modification, especially if that modification is made at some time later when they may have forgotten the original context. It becomes even likelier that they will miss something if they did not write the original piece of code.

In this way, unit tests also provide valuable protection against regressions. They are the building blocks of a well engineered system, and should be seen as a necessary up front cost that will pay off over the lifecycle of a system.

Integration testing

Integration testing can come in many forms, but ultimately it is the practice of verifying that several units or components within a system have been integrated correctly to serve a larger purpose. If we rely solely on unit tests, we only have confidence that the building blocks of our system are correct and we lack confidence that these blocks fit together correctly to fulfill some larger piece of functionality. These tests are more brittle than unit tests and should be used more sparingly, but they are an essential component of a comprehensive testing strategy.

System testing

These tests are high level, and despite the shortcomings mentioned earlier, they are useful in demonstrating that a select number of "happy" and "sad" paths through the system work as expected. When used sparingly in conjunction with the lower level tests mentioned previously these provide an additional layer of confidence that the system as a whole functions correctly in a deployed environment, providing real end user value. Note that within the realm of system testing we also have non functional tests including performance, load, stress, volume and usability testing, but I'll save these for another article.

BDD (behaviour driven development)

While behaviour driven development cannot be considered a testing level as such, it does provide a valuable process for product owners and quality assurance engineers to steer development focus towards end user value. Scenarios are written up front using a DSL that describes the software's expected behaviour. Once the development of a feature is complete, test assertions that satisfy these scenarios are used to verify that the system fulfils this expected behaviour. In my experience, this approach to testing is great at providing shared understanding of the requirements and business confidence in the system.

It's worth noting at this point that despite unit tests being more developer focussed, we should not discount them as a supplementary way to provide business confidence, although some education to the business as to why these are valueable may be required here.

Exploratory testing

Exploratory manual testing should not be discounted, and should be used primarily to test the system in ways that are hard to automate, one example of this being "soak testing". Exploratory testing can also be used to fill in the gaps in your automated tests.

Other testing strategies

This is by no means a comprehensive list of testing techniques, with many other practices providing variable value depending on the nature of a system. These include techniques such as contract testing, infrastructure testing, smoke testing, chaos engineering... the list goes on. It is important to recognise not only where a system is likely to fail, but also where a failure would be most costly, and ensure that adequate testing (preferably automated) is focussed here. It is also important to recognise that a testing strategy is likely to mature alongside a company and a product, i.e. that testing efforts should be proportionate. Too much focus on testing too early on may stifle innovation. Too little focus on testing as the product matures will lead to an increasingly brittle system, increasingly slow development, and an increasing number of bugs and failures in live.

A final word

Don't rest wholly on metrics such as test coverage and number of assertions. Your tests are only as good as the the frequency of bugs and failures in live along with their associated cost to the business. The overall goal of your testing strategy should be to minimise this frequency along with the cost incurred.