As testing is executed by the Applause Community, the testers will be submitting issues when encountered. Through the triage process we – first your TTL then potentially other team members and even you – ensure issues are documented at the right level of detail that will allow your teams to attend, prioritize and ultimately fix them. Over time, trends in the number of issues submitted may draw your attention, specifically when you see a significant decrease. Such shift in the number of issues is likely to also be reflected in your build’s Applause Quality Score and not always for the better, and you may find yourself asking why – and what to do next.
First, it is important to acknowledge that the submitted issues are the symptom, not the cause. It is true that intuitively having less submitted issues hints on less quality problems with the product, however in fact it may also potentially mean that the build was simply not tested well enough, thus increasing the risk of defects escaping into production. Further investigation might be due to identify whether the decrease indicates matters directly related to the implemented testing strategy or the result of improvement in quality of the tested product, website, app or software build.
Here are a few directions to follow up on if you sense that the product quality resulted in the decrease in submitted issues:
- Were there any recent changes to the development process that may have resulted with improved quality? Such changes may be in processes, personnel or tools.
- Was testing executed at a later in the Software Development Life Cycle? Later stage builds are expected to have better quality and see less issues.
- Is the scope of new functionality delivered in the tested build significantly reduced compared to usual? Reduced scope and complexity – such as minor releases and patches – is expected to not only be tested as lesser effort but also to yield less issues.
Obviously, none of these directions might be applicable, and yet the tested product still appears to be less “buggy”. This indeed might be a result of improved quality built into the product. But, in some cases product quality did not change much, yet changes implemented to the testing strategy resulted in less issues than usual.
It is always a good practice to ongoingly revisit and optimize the testing strategy, and is especially important to do so when there is a reduction of submitted issues with no apparent improvement in product quality to explain it. This will increase the confidence in the executed testing, ensure nothing is overlooked and prevent escaping issues.
Here are a few guidelines to follow if you sense that the testing strategy resulted in the decrease in submitted issues:
- Were there any time or other constraints that dictated rushing the build through testing? Allowing the Applause Community sufficient time to run structured and exploratory tests on all applicable areas of your product is key. Note that this may apply more for Exploratory Testing, as tracking the execution of test cases under Structured Testing can be done easily through the Applause Platform.
- Was there a change in the way testing instructions were composed and/or delivered to the testers? Lacking, unclear or confusing instructions not only send the testers to areas in your product you may not need tested, but may also discourage the testers from participating in future testings of your product.
- Are we always testing on the device environments and geographies while ignoring others your product is used in?
- Were the known issues recorded for the product recently updated? An increase of the known issues list – while having a positive impact on reducing “noise” and allowing testers to concentrate on “real” issues – might also result with less issues reported.
Whatever the issue, you are advised to collaborate with your Applause team to further troubleshoot the causes for the decreased number of submitted issues, and identify improvement opportunities in your processes and testing strategy.