As you utilize Applause services to test your products as part of your Software Development Life Cycle, test results are collected and available for you in the Applause Platform. Naturally, a significant portion of your attention will gravitate towards reacting to the collected results – triage issues, correspond with your Applause team to get more information, export issues to your Bug tracking System, and prioritize fixes into subsequent releases. At the same time, you may also like to observe the data in an aggregated, trended manner.
The Product Activity Dashboard provides interactive metrics such as the Applause Quality ScoreTM, tables and graphs, that allow you better understanding of the status and trends of your product and builds. The information – presented at both the product and build level, is geared towards assisting you to:
- Identify the overall quality of testing executed for a build.
- Track trends in quality, and promote process improvement accordingly.
- Focus on areas in your product most requiring your attention.
- Manage the collected issues across the bug life cycle, and acknowledge impact on your team’s tasks.
- Monitor the testing services provided by Applause to allow you and your project team to optimize the testing strategy for your product.
Viewing Product Activity Dashboard
To view the Product Activity Dashboard:
1. Log in to the Applause Platform and navigate to Products.
2. Locate the relevant product from the Products List. You may sort the table, filter it by Status and Application Type, or run a textual search.
3. Click on the Product Name.
The Activity Dashboard for the product will be displayed, with the Product Overview page offering information general to the product, as well as per-build “cards” containing the build’s Applause Quality Score (AQSTM), Confidence Level (CL) and issue distribution at a high level.
Learn more about the Applause Quality Score and Confidence Level here.
4. To drill into the Build Overview page, offering more details about a specific build, click on the “See more details” link at the bottom of the build card.
To optimize performance, not all information displayed is calculated in real time. At the bottom of the Activity Dashboard you’ll be able to see when was the data last refreshed.
The Product Overview aims to provide high-level information on your product, as well as the results collected on your latest builds. Up to 10 builds will be available on a page, and a paging option will be available at the bottom of the page if needed.
Note that the builds are sorted based on their creation date, from the newest to the oldest, allowing you to concentrate on more recent results and trends.
Also note that the metrics displayed are meant to best apply to your needs and implementation. Contact your Applause team if you need to show or hide any of them.
Information displayed about your product might indicate on the level of maturity it achieved with utilization of Applause services, including:
- Date created.
- Number of active components. (Learn more about how to set components to your product here.)
- Number of test cycles previously activated. (Learn more about how to create test cycles here.)
- Bug Tracking System (BTS) integration. (Learn more about how to export issues to your BTS here.)
- Issue backlog management, including Known Issues and Bug Fix Verification. (Learn more about hot to manage your Known Issues list here, and about how to verify fixes for reported issues here.)
You can review and edit the complete details of your product’s settings by clicking on the “See more details” link. Last, you can get a deeper understanding of AQS by clicking on the “What’s the Applause Quality Score” link.
Trends in your testing behavior start to form as you test in the Applause Platform. A recommendation engine uses these trends to notify you – at the build level – on significant variations you should pay attention to. Through the Notifications List you can return to the recent notifications whether dismissed or not, as well as dismiss all notifications at a bulk.
For each of your product builds, the following information will be displayed to provide context to the results:
- Created Date.
- Date when last issue was submitted.
- Number of test cycles previously activated for the build (this also includes cycles where the build was tested only part of the duration).
- Number of active test cycles currently testing the build.
Applause Quality ScoreTM and Confidence Level
The Applause Quality Score (AQS) is a calculated value – ranging from 0 to 100 – which describes the quality of testing results for a product or build during one or more test cycles based on testing done and results collected.
AQS makes it easy to see how your overall quality is trending build-over-build, and to indicate progress and/or opportunity for improvement. The purpose of the AQS is to help make the decisions around when is the right time to release a build much less of a gut call, and more data driven and defensible. By providing a tangible number based on testing results, AQS empowers you to make those critical release/don’t release decisions in a much more fact-based quick and easy manner.
While based on sophisticated data science models, the end results is as simple as a single metric displayed across 3 ranges:
- High: between 90 to 100
- Medium: between 66 to 89
- Low: between 0 to 65
Next to the Quality Score, you will also find an indication of the level of confidence we have in our calculations. The Confidence Level (CL) for an individual build is based on the scale and scope of the testing conducted for the build such as duration and coverage, as well as the breadth of historical data collected on previous builds of the product or app.
Once calculated, the CL is presented as one of three available values:
Learn more about the Applause Quality Score and Confidence Level here.
When applicable, the recommendation engine will notify you on significant variations from your product trends, as recorded while testing your product over time. Click on the notification metric to show more details, or the “What to do?” link to open an article on https://help.applause.com with further explanations and suggestions on what to do next.
After reviewing the notification you may dismiss it by clicking on the “X” to the right. If there are multiple notifications available, you can toggle between them, however note that all of them will be dismissed together.
The Issue Breakdown section provides an overview of the collected issues, throughout the bug life cycle. The total number of issues is displayed across the different phases:
- Submitted: a tester submitted the issue. This count includes pretty much all collected issues, except those discarded or disputed.
- Approved: a project team member (TTL, TE, TA, SDM) or a user from your team triages the submitted issues and approves or rejects them. This count includes all approved issues, except those marked as “Won’t Fix” (you probably care less about those).
- Exported: approved issues may be exported to your Bug Tracking System (BTS) either manually or through the integration. This count includes all issues set with a BTS ID, except those marked as “Won’t Fix”.
You may also break the results on each of those life cycle phases by the resolution status of the issues:
- Not Fixed: fixing the issue is yet completed, or verification for the fix had failed.
- Fixed Not Verified: issue is marked as fixed by the engineering team, however verification for the fix has not yet started / completed.
- Fixed & Verified: the issue was provided a fix by the engineering team, and the Applause community was able to successfully verify it.
The overall fix rate for the build is displayed as well. Learn more about how the test cycle results are managed, including how to export issues to your BTS here.
In addition, the issues are also broken down by their assigned Severity (set by the tester when submitting the issue, and re-confirmed during triage), Value (set by the user approving the issue during triage), and – where applicable – by Priority set in their Bug Tracking System. By default this is presented for the Approved issue – click on the other issue life cycle phases to view the internal Severity/Value/Priority distribution for them.
Last, in case there are some issues still pending triage, this will be indicated. You should complete triage in a timely manner so you can treat the results displayed more reliably (and, it increases the Confidence Level for AQS).
The Build Overview page consists of several sections, providing greater insights into the test results for the build.
To access more detailed data visualizations about the build, click the “See more details” link at the bottom of the Issues Breakdown section of the relevant build.
As in the Product Overview page, high level information about the build is displayed to provide context to the results.
Issue Life Cycle Phases
As in the Product Overview page, you are able to toggle the results by issue life cycle phases, impacting all other Build Overview sections.
As in the Product Overview page, breakdown of the issues by Severity, Value and Priority provides the main context for the testing results.
The Coverage section enables a deeper observation of the collected results based on technology and location.
- Coverage by Technology offers the distribution of results by the Operating Systems issues reported against. To drill even deeper, hover over the Operating Systems (OS) – mobile OSs will be sliced by device and OS version, and desktop OSs will be sliced by web browser and OS version.
- Coverage by Location offers a map view highlighting the distribution of results based on the tester country issues were reported from. You may zoom the map in and out and drag it to focus on interest areas.
Examining the results by coverage allows you to better address issues as well as align the testing strategy of future builds.
Issues by Components
Issues by Components provides an in-depth view of the results by the components set for the tested product. On top of the overall distribution, you may also crosstab the results by:
- Issue Type (default)
- Priority (where applicable)
Identifying how issues are distributed across components allows you to better address issues as well as align the testing strategy of future builds.
The Testing Services section allows you to better understand the testing process. While testing is performed by the Applause Community and driven by your project team, understanding issue accumulations, testing times and triage status keeps you informed about the process, allowing you to better fine-tune the testing strategy of future builds with your Applause project team.
Under the Testing Services section you’ll be able to view:
- Triage, including the overall issue approval rate, distribution of issue approvers between the Applause and your users, and indication of issues still pending triage.
- Testing Times, including average cycle duration and time to first and last issues.
- Issue Accumulation, offering a timeline of the accumulation of submitted issues at the test cycle level.