There are several different ways to define a test case or scenario:
- Atomic test scenario – This is how we automate and what we mean when we say ‘test case’. It’s a self contained, atomic set of steps that produce a specific and measurable expected result. We generally try to have no more than 1-3 assertions per scenario.
- End to end test scenario – A linked set of atomic scenarios that run in a specified and controlled sequence. Typically one large end-to-end scenario may become 2-3 atomic scenarios.
This is important to differentiate, because an atomic test scenario will take us X lines of code and an end-to-end test scenario (which some may just call a test) could be X + 100’s of lines of code and consist of asserting 20-50 conditions.
During the Quick Start, Applause will investigate and correct any script failures (that are not related to application code).
After the Quick Start, investigating and correcting a failure would fall on your team or Applause, depending on what is specified in your Go-Forward plan.
If your plan includes hours for additional script development and maintenance:
- Automation Support Engineers (ASEs) will investigate any script failures that occur during your scheduled runs.
- If the script failed due to a problem with the script, ASEs will attempt to adjust the script and then run the script again. If they are unable to correct the script, they will elevate the issue to the SDET team.
- If the script fails due to an application bug, ASEs will log the issue as a bug which will then get pushed to your bug tracking system and the Automation Dashboard.
For both mobile and web we recommend using a CI server, like Jenkins, to host jobs that run targets in our automation package. Our automation executes these targets and runs a group of tests specified by the job parameters.
Tests are executed using the Applause Cloud, on real devices and web browsers. Work with your Lead SDET to determine which devices you would like your tests run on.
Optionally, customers may choose to utilize the Applause Private Cloud or on-premise devices which provide devices dedicated to your company.
A current list of available devices is available upon request.
Our page object layer makes it much easier and faster than other solutions to maintain scripts. When something changes in your app, instead of updating multiple scripts in several places, there is only a need to update the locators in the page object layer once.
Depending on your post-Quick Start plan, this is where maintenance hours would come into play – our support engineers will be standing by to review any failures, and can fix the page object, which in turn fixes all of the scripts.
Any test scripting work done by Applause is owned by the customer once it is completed.
During the Quick Start, Applause will work with your team to provide the resources and knowledge they’ll need to create new scripts and maintain existing scripts.
Upon the completion of the Quick Start, a Go-Forward plan will be in place which will determine who is responsible for maintaining/adding scripts after that point. This can range from 100% your team (with support/guidance from Applause SDETs) to 100% Applause, where we grow and maintain the solution as a white glove service.
Failures could be due to:
- An application bug
- App evolution
- Locator change
- Test data change
- Device failure
We can utilize your Github instance to store code, however we would need developer access to the repository to do so. If you would like, you can set up a completely separate instance of your repository just for us.
Jobs are groupings of test scripts. Each job will have multiple run IDs (for each time a job is executed). Jobs can be grouped based on:
- Priority (e.g. if you note your tests as P0, P1, etc)
- Component (e.g. “Registration Form”, “Checkout”)
- Feature, or grouping of components
Typically, the Automation Team Lead will define these groupings for you based on your specific app and best practices, unless you already know how you want them grouped.