Performance Analysis in CPI: Generating a Heap Dum...

vadimklimov · ‎2021 Jan 27

Disclaimer

Material in this blog post is provided for information and technical features demonstration purposes only. Although the described technique is based on usage of standard features and capabilities of SAP JVM, imprudent application of operations described below may introduce risks to performance and stability of SAP CPI runtime. Please use the below described technique at your own risk.

Some time ago, I have already written a blog post about memory consumption and garbage collection analysis that can also be applied to SAP CPI runtime. That time, we looked into how to retrieve and make use of SAP JVM garbage collector verbose logs. In this blog post, I would like to expand on the subject of memory analysis and look into another useful technique – analysis of memory snapshots, in particular, memory heap dumps and how they can be triggered in CPI.

Prerequisites

Ahead of reading this blog post further, please ensure that you are comfortable with topics listed below, as they are not a subject of this blog post and will not be covered here, but corresponding knowledge is a prerequisite to make value-added practical use of material described later on:

Memory areas in a JVM. Our analysis will be focused on heap memory, so awareness about particular heap memory areas, memory allocation and deallocation, references between objects, promotion of objects in a heap is essential.

Tools that can be used for parsing, visualization and inspection of heap dump content, and techniques that are instrumental during analysis of heap dumps.

Overview

There are several steps that I'm going to accomplish:

Trigger and generate a heap dump at a CPI runtime node,

Transfer a heap dump from a CPI runtime node to a local environment,

Parse and analyze a heap dump.

This blog post will focus on and go into details about the first step – triggering heap dumps in CPI, and will allude cursorily to other mentioned steps.

Trigger a heap dump in a CPI runtime node

There are two ways how a heap dump can be generated:

Automatically, in the event of OutOfMemoryError that is encountered by the JVM at runtime, when a JVM cannot allocate enough memory that is required for an object. This requires a JVM argument -XX:+HeapDumpOnOutOfMemoryError to be set.

On demand, by triggering a heap dump operation in a running JVM.

Here, we will focus on the second way, as it allows to capture a snapshot of JVM heap memory at any required time.

Before we proceed further, I would like to highlight that an operation of triggering a heap dump is considered to be one of "expensive" operations since it involves generation of a current snapshot of a relatively large memory area and dumping it into a file. This operation isn't intended to be used frequently, but shall only be considered for specific use cases, when precise inspection and analysis of heap area content at a given instant is required.

Several GUI and CLI tools exist that can be used to trigger a heap dump in a JVM, but it is required to have an appropriate access to an operating system where a JVM runs – such as access to an operating system of a runtime node of SAP CPI, and this isn't accessible to customers. What we can do instead is to trigger a heap dump programmatically using built-in JVM functionality that is accessible using a JMX framework. In particular, we are going to utilize an MXBean that provides a diagnostic management interface for the HotSpot VM – HotSpotDiagnosticMXBean. SAP JVM that is used by SAP CPI runtime is no exception here and it provides this bean, which we can access and use for our purposes.

I'm going to develop a Groovy script that will implement the necessary logic, and add the script to an iFlow. The iFlow uses HTTPS sender connection – so that I can call it using an HTTP client to get the script executed, trigger a heap dump and make it generated in a given directory on the CPI runtime node file system.

I’m using a CPI trial tenant in a Cloud Foundry environment for this demo, so I will utilize an existing filesystem hierarchy that a CPI runtime node uses. In particular, a heap dump that will be generated, is going to be placed in a temporary directory that can be found in a home directory of a user who started the JVM of a CPI runtime node – namely, in a directory /home/vcap/tmp.

Using a technique of executing shell commands on a CPI runtime node that I described earlier in my other blog post, we can do some verifications:

Check the current user using a whoami command. Command execution output:

vcap

As seen, the current user is vcap.

Check the current user's home directory using a getent command: getent passwd vcap. Command execution output:

vcap:x:2000:2000::/home/vcap:/bin/bash

As seen, a home directory of the vcap user is /home/vcap.

Finally, we can use a ls command to check that the needed directory exists in its home directory: ls /home/vcap/tmp/.

Now we come to a Groovy scripting part. In sake of simplicity, I will hard-code a path to an output directory where a heap dump is to get generated – certainly, this can be externalized and turned into a configurable parameter. I will also use some additional information (such as a JVM process ID and timestamp of triggering the heap dump) when composing a file name for the heap dump, so that several heap dumps can be generated without a risk of being overwritten.

Below is the code snippet of a Groovy script function for the iFlow that triggers a heap dump when being invoked:

import com.sap.gateway.ip.core.customdev.util.Message

import com.sun.management.HotSpotDiagnosticMXBean



import javax.management.MBeanServer

import java.lang.management.ManagementFactory

import java.lang.management.RuntimeMXBean

import java.nio.file.Files

import java.nio.file.Path

import java.nio.file.Paths

import java.time.Instant

import java.time.ZoneId

import java.time.ZoneOffset

import java.time.format.DateTimeFormatter



Message processData(Message message) {



    String directoryPathName = '/home/vcap/tmp'

    Path directory = Paths.get(directoryPathName)

    String response



    if (Files.exists(directory) && Files.isWritable(directory)) {

        def now = Instant.now()

        String timestamp = DateTimeFormatter.ofPattern('yyyyMMdd_HHmmss').withZone(ZoneId.of(ZoneOffset.UTC.id)).format(now)



        RuntimeMXBean runtime = ManagementFactory.runtimeMXBean

        String jvmName = runtime.name

        String pid = jvmName.take(jvmName.indexOf('@'))



        String fileName = "java_pid${pid}_${timestamp}.hprof"

        Path file = directory.resolve(fileName)



        String hotSpotDiagnosticBeanName = 'com.sun.management:type=HotSpotDiagnostic'

        MBeanServer server = ManagementFactory.platformMBeanServer

        HotSpotDiagnosticMXBean hotSpotDiagnostic = ManagementFactory.newPlatformMXBeanProxy(server, hotSpotDiagnosticBeanName, HotSpotDiagnosticMXBean)

        hotSpotDiagnostic.dumpHeap(file.toAbsolutePath().toString(), true)



        if (Files.exists(file)) {

            response = "Heap dump was saved to file ${file.toAbsolutePath().toString()}"

        } else {

            response = "Heap dump cannot be saved to file ${file.toAbsolutePath().toString()}"

        }

    } else {

        response = "Heap dump was not created. Directory ${directory.toAbsolutePath().toString()} does not exist or is not writable"

    }



    message.setBody(response)



    return message



}

For demonstration purposes, I'm going to make a message processed for another iFlow in parallel, so that it will cause creation of additional objects of a substantial size in a heap area. And while that message is processed, I invoke the demo iFlow and trigger a heap dump:

A response message indicates that a heap dump has been successfully created, and this can be verified by listing content of the directory – for example, using a ls command – ls /home/vcap/tmp/ – one more time:

java_pid12_20210125_193010.addons

java_pid12_20210125_193010.hprof

Note that SAP JVM usually generates two files: a heap dump file (*.hprof) that contains a heap memory snapshot, and an accompanying file (*.addons) that contains additional information about the JVM.

Transfer a heap dump from a CPI runtime node to a local environment

After a heap dump has been generated by CPI, it is needed to be transferred to a location from where it can be parsed by heap dump analysis tools.

One of the most straightforward options would have been to develop a Groovy script that uses File I/O API (or its successors - NIO and NIO.2) to read content of a heap dump file, and then embed the script into an iFlow that can send the content to a required destination (an SFTP server, web server, etc.). While this solution is simple to develop, I strongly discourage its usage: generated heap dump files can be relatively large-sized, and if their entire content is read and processed as a single message in CPI, this can significantly impact performance and stability of a CPI runtime node, or even provoke an out of memory error.
In order to prevent such negative effect, there are several approaches that can be considered – for example:

Split the generated heap dump file in smaller chunks (files) and read / process them one by one in the iFlow, then assemble them back to a single heap dump file after transfer.

Refrain from involving a CPI application into the process of retrieving the heap dump file, but instead use tools and utilities available on an operating system of a CPI runtime node to transfer the heap dump file to a desired destination.

The goal of this blog post isn't to dive deep into these or any other options, so not to digress from the subject, I will only mention here that an approach that I used in this demo, was to utilize an utility scp that is available in many Linux distributions (this utility is available in a CPI runtime node, too) and that is used to copy files between hosts over an SSH protocol – for example, the utility is commonly used for copying files between local and remote machines. I executed an scp utility on a CPI runtime node to copy the heap dump file from its local directory (a directory where a heap dump was generated on a CPI runtime node) to a remote SFTP server. I run the SFTP server on a Raspberry Pi machine and use it as a dropbox. On one hand, the Raspberry Pi machine runs in my internal network, so it is accessible to my local machine where heap dump analysis tools are installed (files that arrive to the SFTP server, can be further transferred to that local machine). On the other hand, it is also accessible from the Internet (hence, a CPI runtime node can access that SFTP server), thanks to incoming requests forwarding setup on an externally facing router. A user account that I used when copying files to the SFTP server, has been configured to support passwordless login – in particular, it has been enabled for SSH public key authentication, which made it possible for me to execute an scp utility in a non-interactive mode on a CPI runtime node.

After copying required files from a CPI runtime node, we need not to forget to do some cleanup and delete no longer needed files from there – for example, using a rm command.

Parse and analyze a heap dump

We have brought a heap dump to a local environment, and now we can utilize familiar tools to parse it – Eclipse Memory Analyzer, IntelliJ IDEA, VisualVM, to name a few. I'm going to use Eclipse Memory Analyzer since this is the tool which have come into common use when analyzing heap dumps of JVMs that are a part of many other products in the SAP product portfolio. Let's open this heap dump in Eclipse Memory Analyzer and let the tool parse its content.

We can now analyze content of the snapshot with the help of some built-in reports, or access a retained dominator tree and navigate through it thoroughly and inspect object references, or select specific groups of objects and explore their properties using the built-in Object Query Language (OQL). Eclipse Memory Analyzer provides many useful tools to analyze the heap dump from different perspectives, and it can be extended with plugins that make its functionality even richer.

For example, overview of the heap dump that we acquired from a CPI runtime node, indicates that there was a single large object that significantly contributed to overall heap consumption:

Using the dominator tree, we can conduct further more detailed analysis of this object and its references:

We can also complement this analysis with checking details of the thread that uses this object (indication of the thread is already seen in the dominator tree above). This all together lets us conclude that the object was created by a custom Groovy script (that was executed as a part of message processing for another iFlow) and a reason why we experienced high heap memory consumption was because the script initialized a large StringBuilder object (which was yet empty at a time of dumping a heap memory snapshot).

Indeed, this is the made-up and simplistic example of the issue that I deliberately prepared for this demo. In contrast, real-life issues that occur and need to get troubleshooted in the area of JVM heap memory utilization and consumption, are very diverse, and root causes can turn out to be much less obvious, so it is usually required to use other features and tools offered by heap dump / memory analysis tooling and more advanced analysis techniques, but I will wrap up for now, as heap dump analysis is a separate topic on its own and its discussion goes far beyond the scope of this blog post.