Event Management Queue Health and Performance Monitoring

[BerndSieger]

Monitoring SAP Event Management Extraction Queues

In previous cases, SAP Event Management (EM) extraction queues in application systems have experienced sudden growth. These queues refer to RFC outbound queues in any application system connected to an SAP EM system (e.g., "EM_TMS_TOR", "EM_ESC_DELIV"), which can be viewed in transaction SMQ1.

Problem Overview:

When this growth occurs, the queue may still appear to be running without errors, and this can lead to confusion. Customers might assume an issue has occurred and contact SAP Support to investigate the sudden growth. In most cases, however, this issue arises when a queue reaches a saturation point: more new entries are created than are processed and removed from the queue.

Two key factors contribute to this situation:

1. The number of new entries created (related to application object changes and event messages sent).
2. The processing time for entries in the SAP EM system (the RFC destination system).

Unfortunately, these variables are usually not visible to users and administrators, and they may not be aware of the queue approaching saturation.

Solution Overview:

To help administrators monitor queue health and detect when a queue is approaching saturation, we’ve introduced a development that provides visibility into the number of calls to the SAP EM system and into the number of processed objects and the time required for processing queue entries in the SAP EM system.

This solution is delivered via two SAP Notes:

• SAP Note 3516253: To be implemented in the application system (where the outbound queues reside).
• SAP Note 3513421: To be implemented in the SAP EM system.

Important: These notes cannot be implemented manually or through the SAP Note Assistant. You must implement the referenced Support Packages.

Key Features of the SAP Notes:

SAP Note 3516253 - Queue Health Monitoring in the Application System:

This note introduces the ability to monitor the "health" of EM extraction queues. Health, in this context, refers to the ratio of idle versus busy times for a queue.

• Transaction /SAPTRX/MONQH: Used to monitor queue health.
  
  
• Queue Name: The name of the queue.
• RFC Destination: The destination system for RFC communication (note that it is case-sensitive).
• No. of days the job runs: Specifies the number of days to monitor.
• Sample Interval (seconds): Determines how often the queue status is checked.

Stopping an active monitoring process is done within the same transaction (use the inactive button ("Stop monitoring process") as shown in the interface).

SAP Note 3513421 - Evaluation in SAP EM System:

This note provides evaluation functions within the SAP EM system and includes start/stop functionality for BAPI performance monitoring.

• Transaction /SAPTRX/EVALQH: Used to evaluate monitoring data from the application system.
  
  
• Application System: The system configured in your SAP EM customizing (configured via transaction SPRO → SAP Reference IMG → Event Management → General Settings → Define Application System).
• Queue Name and RFC Destination: Correspond to the parameters used in the /SAPTRX/MONQH transaction (again: RFC Destination is case-sensitive!).
• Start/End Date: Defines the subset of the monitoring data you are evaluating.
• Idle% Threshold: Defines the minimum percentage of idle time to consider the queue "healthy." The default might fit, but you can adapt it if you either would like to be alerted early or if you want to prevent small spikes to raise alarms.
• Sample% Threshold: Used to exclude days with insufficient sample data.

After executing the evaluation, you can view the results:

• Idle Counter: Counts the number of times the queue has been idle.
• Busy Counter: Counts the number of times the queue has been busy (including error statuses).
• Idle Percentage: Shows the ratio of idle to total samples.
• Sample Percentage: Displays the number of samples taken compared to the maximum number of samples within the evaluation period.

Red fields indicate values lower than the Idle% Threshold. Grayed out fields relate to an insufficient number of samples, compared to the Sample% Threshold.

Data can be exported to a spreadsheet for further analysis or graphical representation.

You can schedule /SAPTRX/EVALQH to run periodically in the background. If any Idle Percentage falls below the set threshold, the job will be marked as "error status," enabling automatic monitoring for potential issues.

BAPI Performance Monitoring:

To gain further insights into queue health, the BAPI Performance Monitoring function can be used. This feature makes previously hidden variables visible, helping administrators perform root cause analysis if queue health deteriorates.

• Report /SAPTRX/ACT_PMON: Starts/stops BAPI performance monitoring.
  
  
• The system records the number of objects processed and the time required for each BAPI call (/SAPTRX/BAPI_EH_POST and /SAPTRX/BAPI_EH_ADDEVENTMSG_02), as well as additional information about the process type and the source of the call.
• Monitoring provides insight into performance, helping to identify bottlenecks.
  
  
• Report /SAPTRX/EVAL_PMON: Evaluates BAPI performance data.
  
  
• You can filter data by BAPI Type, Origin/Sender, and Process Type.
• The Threshold time per object (ms) allows you to define acceptable processing times and highlight underperforming calls. With that parameter you can easily spot low performance in the resulting table:
  
  

Technical Details:

Queue Health Monitoring (Application System):

• The monitoring process is implemented as asynchronous RFC, running at intervals defined by the Sample interval parameter.
• This process is lightweight and should not impact system performance, even when monitoring multiple queues.
• To check the status of the monitoring process, you can use transaction /SAPTRX/MONQH or look for locks on table /SAPTRX/MONQH (via transaction SM21).
• The user starting a monitoring process needs the authorization for transaction SMQ1. The RFC user reading the monitoring data (the evaluation program in the SAP EM system is triggering the RFC) also needs this authorization.

BAPI Performance Monitoring:

• Monitoring is based on calls to the /SAPTRX/BAPI_EH_POST and /SAPTRX/BAPI_EH_ADDEVENTMSG_02 BAPIs. Only runtime data for successful executions will be recorded.
• No DB locking issues are expected, and the impact on system performance is minimal.
• An "empty" BAPI call (e. g. created with the test function of transaction SE37 or with custom code) has zero objects, but for the "Time per Object" calculation it will be treated as having one object.
• There currently is no support for the way an SAP Object Event Repository implementation (SAP EM 9.20 only) will affect the number of objects in a BAPI call. If you are using SAP OER it is recommended to filter out the related Process Types in the evaluation report.
• The processing of repost data from the tables /SAPTRX/RP_* or of event messages from table /SAPTRX/LOCKEDEH (see SAP Note 656990) will not be monitored. The assumption is that any performance changes regarding BAPI executions will affect the processing from those sources in the exact same way.

The volume of the DB tables used for both, queue health monitoring in the application system and BAPI performance monitoring in the SAP EM system, is supposed to stay negligible, due to the aggregation of data on day level.

Irregular Monitoring Cancellations:

• The monitoring process may be cancelled due to system conditions (e.g., application server restart, resource issues). If this occurs, you can use the /SAPTRX/MON_QH_RESTARTER program to restart the monitoring process automatically.
• The program will attempt to resume monitoring based on the remaining days, though precision of the scheduling is limited to one day (-> you might monitor slightly longer than originally intended).
• If there are frequent cancellations, /SAPTRX/MON_QH_RESTARTER should be scheduled as a periodic job.

Conclusion:

These developments allow you to monitor the health of EM extraction queues and evaluate the performance of BAPI calls, providing greater visibility into the system. By implementing the recommended SAP Notes (or rather the referenced Support Packages) and using the associated transactions, administrators can proactively manage system performance, identify bottlenecks, and maintain optimal queue health.