As testing is executed by the Applause Community, the testers will be submitting issues when encountered. During the triage process we – first your TTL then potentially other team members and even you – ensure issues are documented at the right level of detail and their severity and value are properly set to allow you to prioritize and ultimately fix them. In case your Bug Tracking System (BTS) is integrated to the Applause Platform, issues will be exported and handled on your BTS. Based on the integration configuration, issue priority as set on your BTS may be passed back to the Applause Platform through the 2-way integration. Learn more about integrating the Applause Platform to your BTS here and about the 2-way integration with Jira here. Having the accurate priority of issues as set (and often updated) by your teams ensures more accurate reporting in the Applause Platform and specifically an Applause Quality Score (AQS) that better represents your priorities. Over time, trends in the overall priority of issues submitted may draw your attention, specifically when you see a significant increase. Such shift in severity is likely to negatively reflect on your build’s AQS, and you may find yourself asking why – and what to do next.
First, it is important to acknowledge that the submitted issues are the symptom, not the cause. The priority of submitted issues is tightly linked to the quality of the product, both directly (e.g, the actual quality of the tested product, website, app or software build) and indirectly (the way that quality is measured – such as by AQS). It is always important to understand your test results, and it is especially critical to do so when there is an increase in the overall priority of issues. In other words, you will want to know whether the increase in priority is due to actual poor product quality, or due to the testing itself.
Here are a few directions to follow up on if you sense that the product quality resulted in the increase in overall issue priority:
- Were there any recent changes to the development process that may have resulted with poor quality? Such changes may be in processes, personnel or tools.
- Was testing executed at an earlier stage in the Software Development Life Cycle? Later stage builds are expected to have better quality and see lower issue priority.
- Is the increase in overall issue priority attributable to a handful of “P1” issues or to a large number of “P2” and “P3” issues? Clearing your release blockers should have a greater positive impact on the overall priority than fixing the same number of lower-priority defects.
- Similarly, can you attribute the overall issue priority to infrastructural and/or backend changes made recently? It is not uncommon to have even small infrastructural changes resulting with highly-prioritized issues.
Obviously, none of these directions might be applicable, and yet the testing results still indicate on a lower-quality product. In case your product’s quality did not actually change much, it is worth examining if changes to the implemented testing strategy may have resulted in a higher than usual overall issue priority. It is always a good practice to ongoingly revisit and optimize the testing strategy, and is especially important to do so when there is an increase in issue priority with no apparent decrease in product quality to explain it. This will increase the confidence in the executed testing, make you prioritization easier and more accurate, and tighten the relations between your Applause team and your product.
Here are a few directions to follow up on if you sense that the testing strategy resulted in the increase in overall issue priority:
- Was there a change in the way testing instructions were composed and/or delivered to the testers? Testers may inaccurately represent the severity of issues as they document them without a proper understanding of your product, use cases, and their impact. Note that while the severity might be updated to more accurate values during triage, such frequent changes might discourage the testers from participating in future testings of your product.
- Was there a recent change in personnel – specifically in the interface between you and the Applause team – that may have resulted with lost knowledge or misaligned perspectives? Changes are inevitable, yet documentation and knowledge transfer are key to ongoing success.
- Is the scope of new functionality delivered in the tested build significantly different than usual? Increased complexity in the requirements and “big” new features in general, may result in testers misestimating impact of issues.
Whatever the issue, you are advised to collaborate with your Applause team to further troubleshoot the causes for the increased overall issue priority, and identify improvement opportunities in your processes and testing strategy.