Using JVMTOP and JVMMON to analyze Tomcat and Adaptive Processing Server CPU bottlenecks

[Toby_Johnston]

Troubleshooting CPU bottlenecks on the Apache Tomcat application server can be challenging.  Since Apache Tomcat is running as a Java Virtual Machine, understanding which JVM threads are consuming the overall Tomcat CPU can be a bit of a black box since this information is not easily accessible in regular monitoring tools.  Recently, I have discovered a brilliant new tool called JVMTOP which provides direct insight into the performance of threads running on a Java Virtual Machine.  Additionally, JVMTOP also shows in real-time which threads are blocked and also the id of the thread it is currently waiting on.  In this article, I will provide a demonstration on how this tool works and what information you need to collect once you have identified the problematic Java thread(s) so that SAP can perform a deeper root cause analysis.  Note that although this example is based on Apache Tomcat, JVMTOP will also work when troubleshooting CPU issues with the Adaptive Processing Server / Adaptive Job Server (child processes).

Identify the CPU bound thread using JVMTOP

I have made some modifications to the JVMTOP application (updated the executable jar manifest for ease of use) and created a new batch file to launch the application.  Download the version linked from this article when following this tutorial.  Note, if your Apache Tomcat currently uses Oracle JVM, you should configure it to use SAP JVM instead.  For instructions, refer to How to configure Tomcat to use SAP JVM instead of Oracle JVM - Business Intelligence (BusinessObject...

1. Download jvmtop.zip
    
2. Extract the files from jvmtop.zip to <INSTALL_DIR>\SAP BusinessObjects Enterprise XI 4.0\win64_x64\sapjvm\bin on your Apache Tomcat node
    
3. Open a command prompt and browse to <INSTALL_DIR>\SAP BusinessObjects Enterprise XI 4.0\win64_x64\sapjvm\bin
    
4. Open task manager, click view --> select columns, select PID (Process Identifier) and determine the process id of tomcat.exe
    
   
   
   
5. In the command prompt, execute the command: runJVMTOP.bat <PID>.  For example:
   
   
   
   
6. JVMTOP will connect to the SAPJVM and list the top 10 Java threads which are consuming the most CPU.  Notice, it will continue to stay connected and the values are refreshed automatically in real-time similar to the Unix version of top
   
   

   COLUMN	DESCRIPTION
   TID	JVM thread ID
   NAME	Name of JVM thread
   STATE	The state that the thread is currently in
   CPU	Current CPU utilization (in ratio to available cpu time on all processors)
   TOTALCPU	CPU utilization (in ratio to process cpu consumption) since the thread is alive
   BLOCKEDBY	If this thread is blocked, this column shows the thread id which is blocking this thread

   
   
7. In the above example, we can easily identify that the thread creating the CPU bottleneck is http-8080-5 with thread id 48
   
   
   

Create thread dumps using JVMMON

Now that we have identified the offending thread, we need to generate a thread dump so that we can analyze the call stack of the code currently being executed by this thread.  SAPJVM includes a utility called JVMMON which makes creating thread dumps very easy to do without a lot of additional configuration.

1. Open a command prompt and browse to <INSTALL_DIR>\SAP BusinessObjects Enterprise XI 4.0\win64_x64\sapjvm\bin
2. Execute the file jvmmon.exe and it will print out a list of running JVM processes and their corresponding process id (PID)
3. Enter the process id of the Apache Tomcat Server JVM and press enter
   
   
4. Enter the command “print to file” and when prompted for “file=” enter a path and file name for the first thread dump. NOTE: You must first create the folder structure where you intend to save the file.
   
   
5. Next, enter the command "print stacktrace" and at the same time, take a screenshot of the current JVMTOP output. 
   
   
   
   The threads shown in JVMTOP will match the thread details shown in the thread dump and then deeper root cause analysis can be performed by SAP
   
   
   
   
6. Repeat the thread dump / JVMTOP screenshot collection while the CPU is bottlenecked (at an interval of five minutes between each collection).  The more iterations of this information that is collected, the easier it will be to diagnose the bottleneck
   
   
7. Exit JVMTOP by typing the keyboard combination CTRL-C and choose (Y) to end the batch job.  IMPORTANT: Do not click the X to close the command prompt window while JVMTOP is running.  Use the command exit to exit from JVMMON
      
   

     

For more information about JVMTOP, you can visit the homepage at:

Documentation - jvmtop - Java monitoring for the command-line - Google Project Hosting