Technology Blogs by SAP
Learn how to extend and personalize SAP applications. Follow the SAP technology blog for insights into SAP BTP, ABAP, SAP Analytics Cloud, SAP HANA, and more.
Showing results for 
Search instead for 
Did you mean: 
SAP Community Moderator Note: This blog post is outdated. The SAP Cloud SDK for Continuous Delivery and its features have been merged into project "Piper". Therefore we recommend to use the General Purpose Pipeline of project "Piper" instead of the SAP Cloud SDK pipeline. The reasoning for this change, as well as further information on how to adopt the General Purpose Pipeline are described in this guide.



This blog post will guide you through the performance tests stage in SAP S/4HANA Cloud SDK Pipeline as part of the Continuous integration and Delivery (CI-CD) process.

Note: This post is part of a series. For a complete overview please visit the SAP S/4HANA Cloud SDK Overview.


Goal Of This Blog Post

By the end of this article, you will have an overview of the performance tests stage and the configuration involved. You should also be able to analyze the log and identify the cause of the failure of a pipeline execution.

This tutorial will cover the following areas

  1. Introduction to performance tests

  2. Performance tests stage in CI-CD

  3. Performance tests JMeter

  4. Performance tests by Gatling

For a better understanding of this article, we suggest you read the following tutorial first:

Step 14 with SAP S/4HANA Cloud SDK: Continuous integration and delivery

Introduction To Performance Tests

Performance tests are the tests to determine the behavior of the software system and change in responsiveness under simulated load conditions. SAP S/4HANA Cloud SDK Pipeline has integrated support for performance tests using JMeter and Gatling.

Performance Tests Stage In CI-CD

SAP S/4HANA Cloud SDK Pipeline executes performance tests as part of remote tests in parallel to End to End tests of the system. In order to keep the isolation of performance tests from E2E tests, it is necessary to have dedicated servers for each.

The servers where the performance tests are carried out need to be configured in a manifest which can be then used in a stage configuration of pipeline_config.yml. In a performance tests stage, the application is deployed to respective targets that are configured in the manifest before executing tests. An example configuration in pipeline_config.yml is shown below.
#Stage Specific Configurations
- space: 'Perf'
manifest: 'manifest-perf.yml'

Following sections will elaborate the configuration of the performance tests with JMeter and Gatling in detail in an SAP S/4HANA Cloud SDK Pipeline context.

Performance Tests By JMeter

Apache JMeter is an open source performance tests tool which is developed primarily to work with JAVA based applications. One can prepare test plan either manually or by recording it with the help of a recorder. SAP S/4HANA Cloud SDK Pipeline uses a docker image with preinstalled JMeter to execute the tests.

Once the test plans are ready, they can be copied to project directory. If only JMeter is used as a performance tests tool then test plans can be placed in a default location, which is {project_root}/performance-tests directory. However, if JMeter is used along with Gatling, then JMeter test plans should be kept in a subdirectory under a directoryperformance-tests for example./performance-tests/JMeter/.

After placing the test plans in an appropriate directory, JMeter tests can be enabled in the pipeline by placing below configuration in pipeline_config.yml. Where default threshold for failure is set at 80 and unstable is set to 70. Whenever an error rate is beyond these thresholds, the pipeline would result in a failure status.
#Steps Specific Configuration
options: ''
testPlan: './performance-tests/JMeter/*' # mandatory parameter if both JMeter and gatling are enabled
dockerImage: 'famiko/jmeter-base'
failThreshold : 80
unstableThreshold: 70

By default, JMeter is executed in non-GUI mode (-n) and both logs and the dashboard are created as part of the execution. If the performance tests are being executed behind a proxy, the user has the flexibility to configure the proxy by passing them as an option as shown in below configuration excerpt.
#Steps Specific Configuration
options: '-H my.proxy.server -P 8000 -u username -a password -N localhost'
testPlan: './performance-tests/JMeter/*' # mandatory parameter if both JMeter and gatling are enabled
dockerImage: 'famiko/jmeter-base'
failThreshold : 80
unstableThreshold: 70

If the test results exceed the thresholds, the status of the pipeline will be set to unstable or failure based on the threshold. You can find log entry similar to the below example in either case. Log files can be accessed in artifact section with a file name JMeter-report.jtl.
10:57:01 [Performance Tests] Performance: Recording JMeterCsv reports 'JMeter-report.jtl'
10:57:01 [Performance Tests] Performance: Parsing JMeter report file '/var/jenkins_home/jobs/example-project/branches/test-performane-test.luld4o/builds/3/performance-reports/JMeterCSV/JMeter-report.jtl'.
10:57:01 [Performance Tests] Performance: Percentage of errors greater or equal than 10% sets the build as unstable
10:57:01 [Performance Tests] Performance: Percentage of errors greater or equal than 14% sets the build as failure
10:57:01 [Performance Tests] Performance: File JMeter-report.jtl reported 17.4% of errors [FAILURE]. Build status is: FAILURE

If the test results are within the threshold, pipeline status will be set to success and performance tests reports are published in Jenkins along with log files. Log files can be accessed in artifact section whereas reports will be available on the dashboard. A sample JMeter performance tests report is shown below.

As the time of writing this blog post, SAP S/4HANA Cloud SDK Pipeline does not support Remote Testing.

Performance Tests By Gatling

Gatling is yet another powerful and open source performance testing tool for web applications. Gatling performance tests are executed as maven targets, generally as part of the test phase.

One needs to add Scala and Gatling plugins to the maven dependency since Gatling tests plans are written in Scala.

Gatling tests can be launched with a maven command mvn test -Dgatling.simulationClass=yoursimulationname . Where a simulation class is a scala class with a test plan. Like in JMeter, in Gatling too one can either use a recorder or a manual test plan creation. You can find an example test plan below.
import scala.concurrent.duration._

import io.gatling.core.Predef._
import io.gatling.http.Predef._
import io.gatling.jdbc.Predef._

class RecordedSimulation extends Simulation {

val httpProtocol = http
.proxy(Proxy("proxy_host", 8080).httpsPort(8080))
.inferHtmlResources(WhiteList(), BlackList())

val headers_0 = Map(
"Accept" -> "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
"Accept-Encoding" -> "gzip, deflate, br",
"Accept-Language" -> "en-US,en;q=0.8",
"Cache-Control" -> "no-cache",
"Connection" -> "keep-alive",
"Pragma" -> "no-cache",
"Upgrade-Insecure-Requests" -> "1",
"User-Agent" -> "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36")

val scn = scenario("scenario1")
.check(status.not(404), status.not(500))
atOnceUsers(10), // 2

Once the test plans are created, they need to be placed under the performance-testdirectory of the project along with the pom.xml .  Enabling Gatling test in SAP S/4HANA SDK Pipeline is as simple as enabling a toggle in pipeline-cinfig.yml file.
#Steps Specific Configuration
enabled: True

Below is an example log which you will find in the pipeline log when Gatling performance tests are executed.
12:04:39 [Performance Tests] Simulation started...
12:04:40 [Performance Tests]
12:04:40 [Performance Tests] Simulation completed in 1 seconds
12:04:40 [Performance Tests] Parsing log file(s)...
12:04:40 [Performance Tests] Parsing log file(s) done
12:04:40 [Performance Tests] Generating reports...
12:04:41 [Performance Tests]
12:04:41 [Performance Tests] ================================================================================
12:04:41 [Performance Tests] ---- Global Information --------------------------------------------------------
12:04:41 [Performance Tests] > request count 30 (OK=30 KO=0 )
12:04:41 [Performance Tests] > min response time 11 (OK=11 KO=- )
12:04:41 [Performance Tests] > max response time 832 (OK=832 KO=- )
12:04:41 [Performance Tests] > mean response time 196 (OK=196 KO=- )
12:04:41 [Performance Tests] > std deviation 234 (OK=234 KO=- )
12:04:41 [Performance Tests] > response time 50th percentile 105 (OK=105 KO=- )
12:04:41 [Performance Tests] > response time 75th percentile 279 (OK=279 KO=- )
12:04:41 [Performance Tests] > response time 95th percentile 799 (OK=799 KO=- )
12:04:41 [Performance Tests] > response time 99th percentile 828 (OK=828 KO=- )
12:04:41 [Performance Tests] > mean requests/sec 15 (OK=15 KO=- )
12:04:41 [Performance Tests] ---- Response Time Distribution ------------------------------------------------
12:04:41 [Performance Tests] > t < 800 ms 28 ( 93%)
12:04:41 [Performance Tests] > 800 ms < t < 1200 ms 2 ( 7%)
12:04:41 [Performance Tests] > t > 1200 ms 0 ( 0%)
12:04:41 [Performance Tests] > failed 0 ( 0%)
12:04:41 [Performance Tests] ================================================================================
12:04:41 [Performance Tests]
12:04:41 [Performance Tests] Reports generated in 0s.
12:04:41 [Performance Tests] Please open the following file: /var/jenkins_home/workspace/example-project_master-ULMBE43XNJDSWXSCQGCLMWOODKUSREOD5GZIRHSKEQIUSYYAVOEA@4/performance-tests/target/gatling/results/recordedsimulation-1514981078228/index.html

Once the performance tests have been executed the reports can be accessed with the help of a Gatling menu. Which provides detailed insight of results of tests execution including the graphical representation of them. Below images show the performance tests results of an example project.



Performance tests will help you to understand the behavior of the application under simulated load. Thereby helping to understand the implication of new changes on the application before it actually released to production. Integration of Performance tests stage in an SAP S/4HANA Cloud SDK Pipeline is a step towards ensuring better quality production system. The requirement of minimal configuration requirements makes it an easy to use feature.


Further Reading

1 Comment