As testing is executed by the Applause Community, the testers will be submitting issues when encountered. The Severity of the issue is first logged by the tester, and then validated during triage by your TTL and later potentially other team members and even you. Having an accurate estimation of the severity of issues is purposed to help you better prioritize handling them and balancing product quality improvements with resource allocation and deanline commitments. Over time, trends in the overall severity of issues submitted may draw your attention, specifically when you see a significant increase. Such shift in severity is likely to negatively reflect on your build’s Applause Quality Score (AQS), and you may find yourself asking why – and what to do next.
First, it is important to acknowledge that the submitted issues are the symptom, not the cause. The severity of submitted issues is tightly linked to the quality of the product, both directly (e.g, the actual quality of the tested product, website, app or software build) and indirectly (the way that quality is measured – such as by AQS). It is always important to understand your test results, and it is especially critical to do so when there is an increase in the overall severity of issues. In other words, you will want to know whether the increase in severity is due to actual poor product quality, or due to the testing itself.
Here are a few directions to follow up on if you sense that the product quality resulted in the increase in overall issue severity:
- Were there any recent changes to the development process that may have resulted with poor quality? Such changes may be in processes, personnel or tools.
- Was testing executed at an earlier stage in the Software Development Life Cycle? Later stage builds are expected to have better quality and see lower issue severity.
- Is the increase in overall issue severity attributable to a handful of “Critical” issues or to a large number of “High” and “Medium” issues? Clearing your release blockers should have a greater positive impact on the overall severity than fixing the same number of lower-severity defects.
- Similarly, can you attribute the overall issue severity to infrastructural and/or backend changes made recently? It is not uncommon to have even small infrastructural changes resulting with severe issues identified during testing.
Obviously, none of these directions might be applicable, and yet the testing results still indicate on a lower-quality product. In case your product’s quality did not actually change much, it is worth examining if changes to the implemented testing strategy may have resulted in a higher than usual overall issue severity. It is always a good practice to ongoingly revisit and optimize the testing strategy, and is especially important to do so when there is an increase in issue severity with no apparent decrease in product quality to explain it. This will increase the confidence in the executed testing, make you prioritization easier and more accurate, and tighten the relations between your Applause team and your product.
Here are a few directions to follow up on if you sense that the testing strategy resulted in the increase in overall issue severity:
- Was there a change in the way testing instructions were composed and/or delivered to the testers? Testers may inaccurately represent the severity of issues as they document them without a proper understanding of your product, use cases, and their impact. Note that while the severity might be updated to more accurate values during triage, such frequent changes might discourage the testers from participating in future testings of your product.
- Was there a recent change in personnel – specifically in the interface between you and the Applause team – that may have resulted with lost knowledge or misaligned perspectives? Changes are inevitable, yet documentation and knowledge transfer are key to ongoing success.
- Is the scope of new functionality delivered in the tested build significantly different than usual? Increased complexity in the requirements and “big” new features in general, may result in testers misestimating impact of issues.
Whatever the issue, you are advised to collaborate with your Applause team to further troubleshoot the causes for the increased overall issue severity, and identify improvement opportunities in your processes and testing strategy.