As testing is executed by the Applause Community, the testers will be submitting issues when encountered. Through the triage process we – first your TTL then potentially other team members and even you – ensure issues are documented at the right level of detail that will allow your teams to attend, prioritize and ultimately fix them. Over time, trends in the number of issues submitted may draw your attention, specifically when you see a significant increase. Such shift in the number of issues is likely to also be reflected in your build’s Applause Quality Score and not always for the better, and you may find yourself asking why – and what to do next.
First, it is important to acknowledge that the submitted issues are the symptom, not the cause. While intuitively having more submitted issues hints on greater quality problems with the product, it may also potentially mean that the build was simply tested differently than before. Thus, further investigation might be due.
Here are a few directions to follow up on if you sense that the product quality resulted in the increase in submitted issues:
- Were there any recent changes to the development process that may have resulted with poor quality? Such changes may be in processes, personnel or tools.
- Was testing executed earlier in the Software Development Life Cycle? Later stage builds are expected to have better quality and see less issues.
- Is the scope of new functionality delivered in the tested build significantly more complex than usual? Increased complexity in the requirements often result in design mistakes, uncovered edge cases and performance issues.
- Were there any time or other constraints that dictated rushing the build through internal processes while skipping critical steps such as code reviews and unit tests?
Obviously, none of these directions might be applicable, and yet the tested product is still “buggy”. This indeed might be a result of poor quality built into the product. But, in some cases it’s not that product quality deteriorated, but that changes implemented to the testing strategy yielded more issues than usual.
Here are a few guidelines to follow if you sense that the testing strategy resulted in the increase in submitted issues:
- Was testing scope expanded beyond what is executed regularly? Clearly testing more time and/or product functionality increases the probability of finding more defects.
- Was testing performed on device environments and geographies not commonly tested?
- Was testing performed on a new – potentially unstable – environment, such as a newly released mobile device or a beta version Operating System?
Whatever the issue, you are advised to collaborate with your Applause team to further troubleshoot the increased number of submitted issues, and identify improvement opportunities in your processes and testing strategy.