Part 5: E2E Testing integration in CI (Jenkins / GitHub Actions)

[TimoStark]

This is the 5th part of the blog-series about automated testing (end to end and integration-tests) using testcafe. The blog-series has the following parts:

1. Part 1: Overview and first test


2. Part 2: Enhanced Selection, Action and Assertion Possibilities.


3. Part 3: Structure your test and common best-practices


4. Part 4: Code-Coverage E2E tests, Integration Tests


5. Part 5: (this blog) Integration in CI (Jenkins / Github Actions) / Authentification


6. Part 6: Use testcafe for SAP Analytics Cloud and SAP Lumira

Inside this 5th part we will integrate the existing setup into CI tools, and discuss authentification inside an e2e-test.

CI Integration

This step is often underestimated, but extremly important. Writing tests alone does not help - only the automatic execution and handling of failing tests inside your daily work really brings benefit. Unfortunatly there is not "the" single solution for this challange, as the solution mainly depends on what you have already implemented and what CI options you have inside your project / company.

In this blog I will therefore describe the two most common scenarios for E2E && Integration Test CI integration:

1. GitHub Actions: Fairly new (but already almost industry-standard like) way to automate your development workflows provided by GitHub.


2. Jenkins: Pretty old, long term industry-standard, for CI solutions.

While the blog only handles those two integration-solutions, the implementation approach is always the same and can be adapted to any other CI provider (BitBucket, CircleCI, Travis, ...)

As always the complete source code shown inside this blog is available in GitHub.

GitHub Actions

Quoting from the official documentation: "Automate, customize, and execute your software development workflows right in your repository with GitHub Actions. You can discover, create, and share actions to perform any job you'd like, including CI/CD, and combine actions in a completely customized workflow."

What does this mean for us?

• Your code (Frontend or at least E2E) must be inside a GitHub repository to utilize GitHub Actions


• Your GitHub instance (especially for private company instances) must allow / enable the usage of GitHub actions.

Our goal is:

• Execute Integration / E2E tests on every commit.


• In case of any failure, the commit should be marked as failed. Pull Request should be prevented.

Github actions are implemented using a Workflow YAML file in the folder ".github/workflows" inside the root folder of your repository. First create a new file "testcafe-workflow.yml" here.

Let's start with the first lines of this file:

name: End-to-End Tests

on: [push]

jobs:

  test:

    name: Run TestCafe Tests

    runs-on: ubuntu-latest

• name: The Name how the tests will occur inside github


• on: A list of triggers which are causing the Workflow to be started. We will simply run our tests on push. If you integrate your test code inside your frontend repository, every single commit & push will therefore trigger the tests.


• jobs: The jobs which will be triggered inside the workflow
  
  
  
  • test: The name of our job
    
    
    
    • name: description of our job
    
    
    • runs-on: Specify the operation system you want to run your job on. We are taking ubuntu here.
    
    
    
    
  
  
  
  

With this specifiation we have defined the base of our workflow. Now we have to specify the steps to be taken. Testcafe implements an own github-action workflow, which is unfortunatly not really compatible with the ui5-enhancements. Therefore we will take a simple npm runner which is executing a npm script defined in the package.json.

When running tests inside a CI enviroment it is generally recommend to take a headless browser. A headless browser is basically a normal web browser, but without any user-interface, allowing it to run inside a shell on a sever without any graphical interface. Luckily for quite a long time now the most important browser (firefox, chromium based browser) have a headless mode integrated. The start script for testcafe on the server is shown here:

    "scripts": {

//..

        "e2eHeadless": "testcafe \"chrome:headless --start-maximized\""

    },

As you notice the only difference is that we are not starting chrome directly, but "chrome:headless", which (of course) is starting chrome in headless mode.

This script should now be executed by the github actions workflow. To enable this, we are defining the following steps:

1. We are checking out the repository, using the existing github action actions/checkout


2. We have to setup npm / nodejs in order to run our tests. I am using node-version 10 locally, therefore let's also use this on the server


3. We have to run npm install, in order to get the packages available


4. We have to run our npm script, which is triggering testcafe via npm run e2eHeadless


5. We are publising the junit results using an existing junit action.

      - name: Check out the repository

        uses: actions/checkout@v1

      - uses: actions/setup-node@v2

        with:

          node-version: '10'

      - run: npm install

      - run: npm run e2eHeadless

      - name: Publish Unit Test Results

        uses: EnricoMi/publish-unit-test-result-action@v1

        if: always()

        with:

          files: reports/report.xml

This already finalizes our GitHub integration. You can see the Github Action running in the following GIF:

Please remember that this is of course only scratches the possibilities of GitHub actions. Depending of your development workflow I would at least implement a good automated way to inform you about the failing push. Ideas:

• You are using Pull Requests: Prevent Pull Requests from beeing merged, in case of failed tests


• You are using GitHub for issues: Automatically create an issue


• You are using Mattermost / Teams / Slack: Automatically chat to the committer (Mattermost, Teams, Slack )

Juse assume that for any use case you can think of there is already an existing GitHub action - so don't reinvent the wheel here 🙂

Jenkins

Again quote from the Jenkins homepage: Jenkins is an open source automation server which enables developers around the world to reliably build, test, and deploy their software.

What does this mean for us?

• You should have jenkins in place (or any other comparable CI solution)


• Jenkins should have connectivity to your source code repository (GitLab, GitHub, BitBucket, [...]) to be able to react on code changes via WebHooks

In this scenario we will only implement the Jenkins job, based on a Multibranch pipeline job. As a complete tutorial for Jenkins would be too much for the scope of this blog, I will only explain the content of the used Jenkinsfile. You can of course re-utilize this content in any freestyle or scripted pipeline based job.

First we have to install a few plugins (as always when working with jenkins 🙂 😞

• NodeJs. After installation, please configure a NodeJs installation inside the Global-Tool-Configuration with Name "NodeJs" and version 11.


• Credentials-Binding: If you have any credentials / user name / password


• AnsiColor: Colorize the output of your testcafe log


• Testcafe Jenkins Plugin: The official plugin by testcafe. There is also a wonderful more detailed blog describing the integration of jenkins with testcafe.

As there is an open testcafe plugin we have the benefit that screenshots and videos are actually published and visible in jenkins. In order to utilize this functionality we again need an own run script, as we have to use a specific jenkins reporter. The "-r" command line argument is specifiyng the reports and overwriting any setting you've maintained in the testcaferc.json file.

"e2eJenkins": testcafe \"chrome:headless --start-maximized\" -r spec,jenkins:report.xml

The jenkins file itself is very straightforward again:

• Enable usage of NodeJs, and ansicolor


• Disable concurrent Builds, to avoid using too many server resources


• Define one step, which is executing the npm commands to install and run the test


• Afterwardwas (always - also of course when the tests are failing) publish the test data using the TestCafePublisher.

pipeline {

    tools {nodejs "NodeJs" }

    agent any

    options {

        ansiColor('xterm')

        disableConcurrentBuilds()

    }

    

    stages {

        stage('Run E2E tests') {

            steps {

                nodejs(nodeJSInstallationName: 'NodeJs') {

                    sh 'npm install'

                    sh 'npm run e2eJenkins'

                }

            }

            post {

                always {

                          junit keepLongStdio: true,

                          testDataPublishers: [[$class: 'TestCafePublisher']],

                          testResults: '*.xml'

                }

            }

        }

    }

}

After a failed execution the Jenkins Log will include screenshots and videos recorded during the test-run:

 

As for Github actions there is a ton of other plugins available. You should especially invest some time into communicating the failed build to your developers:

1. Push back to your repository (GitLab/GitHub/..)


2. Via Mattermost/Slack/Teams/..


3. Automatically create a Jira issue

Again there is basically a plugin for everything, so again don't reinvent the wheel but use existing plugins.

Authentification

Until now we made our life very easy by simply using an unprotected application without any authentification. Of course in your real life you will probably have a fiori launchpad or any other portal with some kind of authentification. Here two questions arise:

1. How to login?


2. Where to store username and passwords?

Login to Fiori Launchpad

If you are using a simple user-name/password protection, you can access the Fiori-Launchpad User-Name && Password using the known API. The only special in the following code sample is the { anonymize : true } option inside the password input, which will avoid logging the password into the console.

await ui5Action.typeText(ui5("User-Name Field").domQuery('#USERNAME_FIELD-inner'), userName)

      .typeText(ui5("Password Field").domQuery('#PASSWORD_FIELD-inner'), password, { anonymize: true })

      .click(ui5("Login Button").domQuery("#LOGIN_LINK"));

To make your life easier, the overall launchpad login process was encapsulated in a reusable function (ui5Launchpad.startup). You can also directly specify a tile-hash which you want to open at the beginning of the test.

await ui5Launchpad.startup({

         user: user.user,

         password: user.pw,

         tile: params.tile,

        });

If you are using two-factor authentification, I would suggest that you are checking if two-factor authentification is really required for test-users. Having a 2-factor auth. in every single test will slow down your test, increase complexity and also normally does not bring any benefit. From security side, you can argue that the two factor authentification can e.g. only be deactivated for a specific server-ip (your Jenkins/CI instance). If you really have to implement 2-Factor authentification inside your test, you can use the full-power of NodeJs and existing solutions. It is e.g. possible to send a receive SMS/Calls/... using NodeJs (e.g. using Twilio), or for HOTP, TOTP based authentification you can use speakeasy npm package.

Secure your passwords

As always even test code shouldn't contain any real authentification information. Therefore please avoid having the password hardcoded inside the startup/login script. Instead please use the provided API for storing user-name/passwords by your CI provider and pass the username / password as enviroment variable inside your test. On your local computer (for developing purpose) you can set these enviroment variables inside your local machine settings.

• Jenkins: Credentials-Binding: If you have any credentials / user name / password


• GitHub: use secrets

As always: do not implement encryption of usernames / passwords on your own. Always reuse existing / well-established solutions of the tools you are using.

 

In the last blog of this series we will have a look on how to use the ui5-enhancements for testcafe outside of the world of Fiori-Launchpad. Especially we will focus on SAP Lumira, SAP Analytics Cloud and SAPUI5 Web-Components. As always I am happy for any feedback.