Cypress Parallel Testing

[dpasciak]

Overview

I hope that everyone already knows the value of test automation and how much time it may save during new feature release. I agree that we have to develop these tests first, but when they are done, they provide a lot of benefits to daily work. What If I said that there is a way to improve test automation by running them parallelly? This blog describes  two different ways to run Cypress tests in parallel.

What is Parallel Testing?

Parallel testing refers to running tests simultaneously across multiple threads of processes. This approach enables tests to execute concurrently rather than one after another in sequential order. In this method, tests are divided into separate groups or batches and distributed among different threads to run independently and in isolation. This method significantly reduces the total test execution time, as it is divided among the available threads. Increasing the number of threads can further accelerate the overall execution process.

Sequence testing vs parallel testing

To demonstrate how much time can be saved thanks to parallel testing, four simple tests were executed concurrently. The image below shows the achieved results. Dividing the tests into four threads allows to save 36% of the time that would have been required to execute these tests sequentially. In this case ~9 seconds it's not a lot but this is just an example. We should look more broadly and think about how much time parallel testing may save when we have to run thousands of tests twice a week.

Parallel tests in cypress

Local machine

To run tests in parallel on the local machine, we can use the open-source library: cypress-parallel. It may be installed by the following command:

npm i --save-dev cypress-parallel

The most popular way to generate a report based on tests results is provided by cypress-mochawesome-reporter. It can be installed by:

npm i --save-dev cypress-mochawesome-reporter

In the next step, we need to configure a reporter for cypress-parallel. We want to generate a report based on  the result from each thread. First, we will make some changes in the file: cypress.config.js

For sequentially tests run in setupNodeEvents we need to add

require('cypress-mochawesome-reporter/plugin')(on)

But for cypress-parallel, this section should look different:

 setupNodeEvents(on, config) {
      on('before:run', async (details) => {
        await beforeRunHook(details);
      });
    }

Above defineConfig, we need to define constant beforeRunHook 

const {beforeRunHook} = require(“cypress-mochawesome-reporter/lib”)

In the last step, we will create scripts in our file named: package.json It will make the start procedure more comfortable.

  "scripts": {
    "cy:run": "cypress run",
    "cy:parallel": "cypress-parallel -s cy:run -t 4 -d 'cypress/e2e/parallel-tests/*.js' -r 'cypress-mochawesome-reporter' -o 'cypressParallel=true, saveJson=true, reportTitle=Sample Parallel Title'",
    "clean": "rimraf cypress/reports",
    "generate-report": "generate-mochawesome-report",
    "cy:e2e:parallel": "npm run clean && (npm run cy:parallel || true) && npm run generate-report"
  }

Description:

• cy:run - just a shortcut
• cy:parallel - shortcut for cypress-parallel but with additional parameters: 
  • -s Our npm Cypress command.
  • -t Number of threads.
  • -d Cypress specs directory
  • -r Reporter to pass to Cypress
  • -o reporterOptions - options used by a reporter to generate index.html. We can customize report for our personal needs. Available options are described here.

         Please refer to the official documentation for the rest of the parameters

• clean - used to clean our reports folder before the next run
• generate-report - used to generate report
• cy:e2e:parallel - complex command which connects all of the necessary needed for each run

Now to run our test we only have to run the below command:

npm run cy:e2e:parallel

The above quick guide was created based on official documentation of linked libraries and this example. Additionally it shows that parallel test configuration doesn't require much effort and allows us to save a lot of hours during testing.

Multiple machines in Continuous Integration

Continuous Integration (CI) -  involves automating the process of merging code changes from various contributors into a unified software project. It enabling developers to regularly integrate their updates into a shared repository. Once integrated, builds and tests are automatically triggered to ensure the new code is functional and compatible. This process relies on automated tools to verify the correctness of the code before it becomes part of the main project. Cypress tests can be configured to run using platforms such as GitHub Actions, GitLab, or AWS CodeBuild. For detailed instructions and more examples, refer to the Cypress documentation.

Using Continuous Integration (CI), we can run our tests on multiple machines and browsers simultaneously, making it an excellent choice when our project includes a large number of tests. Each CI has its own configuration steps, so it is necessary to refer to documentation specific to the platform that we are using.

Cypress Cloud

Is a dedicated app that provides many features to improve daily work with tests run in CI.

Benefits

• View and debug past test results from your CI environment.
• Analyse and diagnose test health.
• Reduce test time and save money by smartly orchestrating future runs across multiple machines.
• Integrate with your favorite developer's tools to help streamline your workflows.
• View CI runs and test health directly from the Cypress app.
• Configure Cypress Cloud how you want and enable enterprise single sign-on.

For details about each benefit review documentation.

Parallel testing benefits

• Saves time and money.
• Scalability – On a local machine it is limited by hardware capabilities, but in CI as test suite grows we can add additional machines.
• Tests are isolated.

Best Practices

• Proper test data management – Each test case should have its own data for test, and this data should be independent of the other tests. Otherwise, it could have a negative impact on results when you are running your tests in parallel.
• Independent tests – While creating your test avoid dependencies as much as possible.
• Debug test which fails –  Don't rerun all of the tests again but instead debug the specific tests that failed
• Test multiple browser – Run tests across multiple browsers if CI is used it can be done simultaneously. Check how to setup here.