A lecturer at the university had once said to me that you MUST test every single line of code. This would result in a test coverage of 100%. I already asked myself at that time, whether this makes sense?
But where is the right balance in software testing? What is too little? What is too much? That's the question we'll try to answer in this blog post.
But before we get to that, we first want to look at some testing procedures and best practices. Finally, we will show an exemplary development process with different testing procedures.
Test Procedures
In the software industry various test procedures are used to test software in different ways. In this article, we will not go into each individual test procedure, but will point out some aspects that are important when selecting test procedures. Nevertheless, in order to provide an overview, we have listed some test procedures here.
Selection of the Test Procedures
Each test procedure covers a very specific part of the testing. For example, unit tests are used to test individual software components in isolation, and integration tests are used to check the interaction of these components. Different test procedures should therefore be used complementary to improve the coverage. One test procedure alone is not sufficient.
Depending on the test procedure, the software is also viewed from different angles. This is used in white box and black box testing. In black box testing, the tester does not know the internal design of the software. In white box testing, the tester knows the design or the tester is a developer himself. Each person has a different understanding of the software and therefore tests differently. It is therefore important to select the test procedures in such a way that several people test in order to obtain different opinions.
Quality over Quantity
The selected test procedures are either executed manually or automated. Especially with automated execution, the fallacy often arises that the goal has been reached as soon as all tests have been successfully run. But what was actually tested?
The quality of the tests is crucial and should be weighted higher than the quantity. Especially with more complex test procedures, such as penetration testing or load testing, the extended circumstances must be taken into account. It therefore makes sense to define the conditions and the expected result before testing. This is the only way to verify whether the goal has really been achieved.
Supporting tools can also help to further improve the quality of the tests. Static code analysis tools, for example, can be used to find possible sources of errors and automatically determine test coverage. Automated bots can also be used in load tests or penetration tests to find further issues. In addition, artificial intelligence will provide new opportunities for testing.
A high quantity of tests can even become a problem in the long run. If the program code changes, the associated tests must also be adapted. If there are a lot of tests, their adaptation can take a lot of time and ultimately also affects the speed of development.
Team Dynamics
Software development teams come in different sizes and constellations with different practices. In some companies, multiple software teams work continuously on a common product. In other companies, team members work mostly alone on one or more customer projects. Each team member has a different level of education. For this reason, no general statement can be made regarding suitable test procedures for a team.
Suitable Test Procedures
All team members should be familiar with possible test procedures. However, not all test procedures are suitable for every team member. For example, it is not advisable to use a beginner for system testing, as he or she is likely to have less experience in recognizing possible dependencies and resulting errors. Instead, a beginner would be better suited for smoke testing or regression testing. It is therefore important to introduce suitable test procedures that support the team and do not place an additional burden on it.
Responsibility for Quality
Selected test procedures must be integrated into the development processes and lived. As a rule, the entire team bears responsibility for compliance with the processes and the quality of the software produced. However, if shared responsibility does not work, a single person can be held responsible for quality. Despite testing and accountability, errors can occur. This provides an opportunity to improve testing procedures and adapt existing processes.
Everybody has to start somewhere
It is advisable to start small. A good test coverage with unit tests and an additional user acceptance testing or exploratory testing can work wonders. Such an approach is far more effective in the long run than multiple testing procedures that are only partially lived.
Investment and Profitability
We can test all day and still have bugs in our software. There is no guarantee of error-free software. Therefore, risk management is needed to estimate the probabilities of occurrence and consequences.
Risk Management
Testing should always be considered as an investment to minimize risk. The question should therefore be: What errors can we tolerate without causing loss of customers, revenue, image, or cause other damages? How much testing do we need to do to achieve this? Answering this question is not an easy task and requires experience, but also common sense.
Different functional areas of the software may also receive different risk ratings. The core functionality of the software usually carries a higher risk and needs to be tested better than other functionality.
Costs
The various test procedures are executed at different points in the development process. Unit tests, for example, are created during development or, with test-driven development (TDD), even before development. In this way, errors can be detected and corrected at an early stage. Other test procedures, such as usability testing, only take place later in the development process. If an error is not found until usability testing, this can cause high costs and delays in development. So the earlier in the development process the error is found, the better.
Practical Example
Sometimes you just need a starting point. Here you will find a modeled development process with different test procedures. The test procedures can be executed either manually or automated. Note that not all tests have to be performed by the same developer.
UI Tests: Visually tests the changes made to the user interface.
Unit Tests: Tests the functionalities and components in isolation from each other. A code coverage check can be used to define how many lines of code as a percentage of the entire code must be run through by the unit tests.
Integration Tests: Tests the interaction of the components on a test environment. The environment should be as similar as possible to the production environment.
Regression Tests: Tests the core functionality of the software on a recurring basis to ensure that no undesirable side effects occur.
Conclusion
Experience is needed to successfully introduce various testing procedures into the development process. The balance between too much and too little testing is significantly influenced by risk management.
Try out different testing methods and decide which ones work best. Take the team dynamics into consideration and continuously adapt the processes to optimize them.