Open Category List

Managing the Results of Test Cycles

10 minutesread

As the test cycle is being executed and testers are exploring for issues and/or running test cases, results are collected and available for you in the Applause Platform.

Managing the results will allow you to:

1. View the reported issue, test case run and/or review results.

2. Triage the results to ensure only complete, relevant results are approved. Triage will also include rating of the value and severity of reported issues. Should you need additional information, you may also communicate with the project team in a secured, documented channel.

3. Export results to a .csv file for further analysis and/or manual upload to other systems

4. Optional: depending on your implemented workflows, you may also:

  • Mark issues as “Won’t Fix”, “Known Issue” or “Resolved”.
  • Export your approved issues to your Bug Tracking System (BTS), such as Jira.
  • Initiate  Bug Fix Verification (BFV) Workflow

Results are generally organized in the Applause Platform within the test cycle they have been collected in. For Issues, however, there is also a global list, allowing you to manage them across multiple cycles.

Viewing Results

In order to view the test cycle results,  follow the below steps:

1. Log in to the Applause Platform and navigate to Test Cycles.

Test Cycles

The Test Cycles page will be displayed. Basic information on the test cycle results is already included in the list view:

  • Test cycle status
  • Total number of issues, and of which, number of new and pending issues
  • Total number of passed, failed and in progress test case results, and of which, number of test case results awaiting approval

Test Cycles Information

2. Search for your test cycle in the list. You may use the search box at the top, show/hide Closed cycles, and/or re-sort the list by clicking on the column headers. 

If you are logged in to an Agency account, you may also select to show/hide all agency test cycles.

3. Click on the test cycle name to view the complete result data.

Within the Test Cycle page, results are organized in the following tabs:

  • Summary
  • Issues
  • Test Case Results
  • Test Case Coverage
  • Reviews
  • Known Issues

Summary

The Summary tab contains aggregated visuals on bug distribution by component, type and environment, as well as on triage progress.

Issues

The Issues tab contains the list of issues reported by the testers during the cycle, their status, severity & value rating, BTS identifier (if exported) and BFV status (if verified). You can filter the list by a variety of fields, show/hide issues not exported to BTS, search by issue ID (in Applause or BTS, if exported), or run a textual search.

Issues List

Click on the issue title to view the complete Issue Report, where you can view additional information about the issue, and apply different actions on the issue. Learn more on how to work with issue reports here.

You may also apply many actions on multiple issues at a bulk, by selecting them from the grid, and selecting an action from the Actions menu above the issues list.

Note: A similar issue list is also available outside the context of the single test cycle. Navigate to the Issues menu to open the global list of issues and view (and apply actions) on all available issues.

Issues Menu

Test Case Results

The Test Case Results tab contains a list of test case runs executed by the testers and their results, status, test case, environment, executing tester. 

Test Case Results

Click on the test case result ID to view the complete test case result, where you can view the complete information on each test case step, as well as triage the reported result.

You may also apply a few actions on multiple test case results at a bulk, by selecting them from the grid, and selecting an action from the Actions menu above the issues list.

The Test Case Results tab will only be available if the test cycle was set with test cases.

Test Case Coverage

The Test Case Coverage tab contains the breakdown of executed test cases and their results across different environments. This allows you to understand what functionality (test cases) and technology (environments) were covered by the testers during the cycle.

Test Case Coverage

The Test Case Coverage tab will only be available if the test cycle was set with test cases.

Reviews

The Reviews tab contains a list of product reviews provided by the testers, and their overall rating, status and providing tester.

Reviews

Click on the review ID to access the complete review, where you can read through all open-ended and rating answers provided by the tester, as well as triage the provided product review.

Product Reviews

Known Issues

The Known Issues tab contains a filtered issue list, including only issues marked as “Known Issue”. You may apply all other functionality same as in the Issues tab.

Triaging Results 

In order to complete the work on the cycle, all results submitted by the testers need to be triaged, and eventually either approved or rejected. The triage is a critical task, impacting both you and the quality of results you’ll be getting from Applause, as well as the testers’ rating (and payout). Ultimately, the triage is all about ensuring only approved, complete and accurate results will be available for later reporting, export, etc. 

Triaging Issues

For issues, triage may include the following tasks:

  • Edit the issue if needed so it is at par with your content expectations
  • Ensure the issue severity is properly set
  • Leave notes to the Testing Team Lead and the PM if needed
  • Message the tester, in case you require further information on the issue. Testers will be given time to respond, yet the issue will be automatically rejected when the test cycle will be closed by your Testing Team Lead or PM.
  • Review the issue report content, and approve or, if it is invalid, out of scope, or the tester didn’t follow instructions, reject it

Note that your TTL is able to assist you with initial triage, and per your request either recommend approval/rejection, or complete those instead of you.

Please note: Triage is commonly a one-time activity, afterwhich the options to rate, approve of reject a result will be grayed out and inaccessible.

Best practices for issue triage are available.

To triage issues:

1. While in the Issue Report page, click on the Actions button.

Triage Issues

2. Select one of the following:

  • Edit Issue: apply changes to the issue report details. Make sure to save your changes.
  • Approve: approve the issue as a valid one. You will be asked to then set the issue value: somewhat, very or exceptional.

Approve Issue

  • Approve as Won’t Fix: Approve the issue for the sake of tester rating and payout, yet there is no intention on fixing it. The issue value will be automatically set to ‘somewhat’.
  • Reject: reject the reported issue as invalid. You will be asked to set the reason for rejection: duplicate reporting, need more info, tester did not follow instructions, issue is out of defined scope, scenario works as designed, or any other reason. You are encouraged to still add this issue to your Known Issues list to prevent future reporting of it.

Reject Issue

  • Request More Information: the issue report is lacking or unclear, and you’d like the tester to update it.

Request More Information

Triaging Test Case Results

You should also triage the test case runs executed by the testers.

To triage test case results:

1. While in the Test Case Result page, click on the Actions button.

Triage Test Case Results

 2. Select one of the following:

  • Mark as Valuable:approve the test run result as a valid one. 
  • Mark as Unsatisfactory: rate the test run result as invalid.

Alternatively, you may also approve the test case results at bulk from the Test Case Results page by selecting the relevant test case results from the list. Then, click on the Actions button above the list, and select Approve.

Bulk Approve

Click on the Discussion tab to message the tester, in case you require further information on the provided review.

Triaging Reviews

You should also triage the product reviews provided by the testers.

To triage product reviews:

1. While in the Review page, click on the Actions button.

Triage Reviews

2. Select one of the following:

  • Mark as Valuable: approve the review as a valid one. 
  • Mark as Unsatisfactory: rate the review as invalid.

Note that you can also message the tester, in case you require further information on the provided review

Further Managing Issues

For Issues, you may also manage the following attributes:

  • Fix worthiness: mark/unmark the issue as “Won’t Fix”. Won’t Fix issues will be tracked to prevent testers from re-reporting the issue, yet there is no intention on fixing it.
  • Known Issue: mark/unmark the issue as a “Known Issue”. Known issues will be tracked to prevent testers from re-reporting the issue, and to allow a fix verification once fixed.
  • Resolution: mark/unmark the issue as “Resolved”. Resolved issues are considered fixed and verified, thus no need to keep tracking them as known issues.

To further manage issues:

1. While in the Issue Report page, click on the Actions button.

Manage Issues

2. Select one of the following:

  • Mark (or: Unmark) Won’t Fix: the label “won’t fix” will be displayed next to the issue status as well as to the Resolution status.

Resolution Status

  • Mark (or: unmark) as Known Issue: Known Issues are marked by a small tag next to the issue name, and are included in the list under the Known Issues tab…

  • Mark (or: unmark) as Resolved: the Resolution status will be updated.

Resolved

Note: you can also the below at bulk from the issue list in the Issues tab. Simply select the relevant issues from the list, click on the Actions button above the list, and select the required action.

You can read more on how to work with issue reports here.

Exporting Results to a .CSV File

In certain cases, you may want to export the results to an external .csv file. The file may be then used to upload into a different system, recordkeeping, and/or for further analysis.

Generally you may export multiple results – issues, test case results and reviews – at a bulk, by selecting them from the grids of their respective lists, and selecting the Export action from the Actions menu above the list.

Learn more how to export results to external file here

Export to External File

 

Export Test Case Results

Exporting Reviews

Performing Advanced Tasks on Results

Exporting to BTS

If your product is set with a Bug Tracking System integration, you can export the issue to your BTS. Alternatively, if the integration was not set, or if the issue already exists on your BTS, you may simply manually set the ID of the matching issue on your BTS.

To export issues to your BTS:

1. While in the Issue Report page, click on the Actions button, and select Export.

Export to BTS

Note: you can also the below at bulk from the issue list in the Issues tab. Simply select the relevant issues from the list, click on the Actions button above the list, and select Export.

2. In the opened Export Issues dialog, select “To My Bug Tracking System”, and click Export.

Export to My Bug Tracking System

Once the issue is exported, the issue ID as it set in your BTS will be displayed.

For Jira connectors (only), the BTS ID will serve as a hyperlink, allowing you to open the Jira issue directly from the Applause Platform.

Learn more how to export issues to your BTS here

Learn more how to integrate the Applause Platform with your Bug Tracking System here.

To manually set the BTS ID:

1. While in the Issue Report page, click on the Actions button, and select Mark Manually Exported to BTS.

Mark Manually Exported

2. In the opened Manual Export to BTS dialog, set the BTS Issue ID, and click Save.

Manual Export to BTS

Note that the Applause Platform will enforce unique BTS IDs across all issues reported on a specific product, regardless if were exported manually or through the integration.

As you will not be able to export the issue through the integration once marked as ‘Manually Exported’, you can un-mark it. You may also do this in case you need to correct a typed-in error.

Initiating BFV

Once the reported issue is fixed, you may want to utilize the testing community to verify the fix, as a Bug Verification workflow. You may initiate the BFV in an existing or a standalone test cycle. Learn more how to initiate the Bug Fix Verification workflow and differences between BFV methods here.

To initiate a BFV in a standalone test cycle:

1. While in the Issue Report page, click on the Actions drop down, and select Request Verification.

Initiate a BFV

2. The Fix Verification dialog box will open.

Fix Verification

3. Enter the following in the dialog box:

  • Choose a build to conduct the retest on.
  • Type in any comment or further instruction
  • Click Send

Additionally, each issue has a checkbox to the left of it, which allows you the ability to perform batch actions, available under the Actions dropdown menu in the upper left part of the screen. By selecting each issue you wish to modify, you’ll be able to perform the following actions:

  • Export: The Export action allows you to perform a batch export of each issue you’ve selected to either a .csv file, or to your bug tracking system (if configured)
  • Send Issue for Verification: This action gives you the ability to provide the tester with a URL or an updated build which you feel has resolved the reported issue, and requests a fix verification from the reporter of the issue
  • Mark as Resolved: When an issue has been fixed, marking it as resolved will effectively close the issue in the Applause database. Additionally, issues pushed to a Bug Fix Verification will automatically be marked as resolved.
0
0
553
10 minutesread

Related Knowledge Base Posts