BiWhy – WEBI Memory Thresholds and Host Protection

[ilya_ulyanov]

Disclaimer:
BiWhy is once again available for free to everyone at biwhy.net.

Use Case

BOBJ Sizing Guide:
“Enable Memory Analysis” is now unchecked by default when you perform a new installation from BI 4.2 onwards... We now recommend this is disabled as your default setting, enabling the process to use as much memory as it needs and without the need to reclaim memory or generate any related error due to memory limits becoming breached.

My strong recommendation is to keep it enabled in order to protect host memory from being exhausted by Web Intelligence.

Memory exhaustion can happen for two reasons:

Memory leak

I once saw Web Intelligence consume 100 GB during an Excel export. I am sure that specific issue was fixed long ago, but nobody can guarantee that the code is fully leak-free.

Real load

• Multiple concurrent users
• A few users running very large reports
• A combination of both

Think about Memory Thresholds the same way you think about the Java Xmx parameter.

If thresholds are reached, the system will handle the situation gracefully. In the worst case, a Web Intelligence server will restart, affecting only the users assigned to that specific server.

If thresholds are not set, the same scenario may cause the entire host to run out of memory, which can lead to a system-down situation or worse.

Conservative formula for calculating memory allocation for Web Intelligence

WEBI_MAX = Available_RAM – Sum of Xmx (APS and others) – 2 GB for OS – 2 to 4 GB for CMS (or more, depending on actual usage) – Xmx for Tomcats if they run on the same host – other major memory consumers on the server.

Recommendation

Set thresholds high enough that they are never reached during normal operation, but still low enough to avoid endangering host memory.

Why does the Sizing Guide recommend switching off Memory Analysis?

I do not know for sure, but the most likely reason is related to threshold configuration.

Thresholds must be customized for each system. They cannot be set correctly by default, because default values would be either too high or too low.

BOBJ administrators need to understand what the Lower, Upper, and Maximum Thresholds mean, and how to connect the warning messages seen by end users to these configured parameters. That way, they can adjust thresholds or scale up the system instead of opening incidents.

A possible improvement on the SAP side would be to state explicitly in warning messages that they originate from the Memory Analysis configuration.

Threshold meaning from the Admin Guide
If this property is enabled, then the following properties will be active and recognized by the server:

• Memory Maximum Threshold
• Memory Upper Threshold
• Memory Lower Threshold

If the Memory Lower Threshold limit is reached, the server swaps out inactive documents onto the hard disk, allocating additional memory for documents that are active. Each user is allowed to have up to one active document instead of Maximum Documents per User.

If the Memory Upper Threshold is reached, the following server actions will take place in order to free resources and protect the server:

• The server will reject new connections and new client calls. Only the option to save Web Intelligence documents will be allowed. Users requesting an action will receive a Server Busy message and will be notified that they should save any pending changes.
• The server will turn on system cleanup to free enough resources so that the amount of allocated memory is below the limit set by the Memory Upper Threshold property.
• The server tries to close read-only documents.
• If not enough memory is freed during system cleanup, the server will begin to close documents that are in editing mode. The server will begin to close documents based on the LIFO protocol: the most recently active document will be purged from memory first. The server will continue to close documents until a safe level is reached. This level is based on the following calculation: Memory Upper Threshold - (20% × Memory Upper Threshold). For example, if the Memory Upper Threshold property is set to 4.5 GB, then the safe level would be 3.6 GB.
• The server cannot close documents when a client call is running. Any document being refreshed, exported to another format, or involved in another time-consuming operation will not be closed when the server reaches this threshold. If the server cannot recover enough memory and remains above the Memory Upper Threshold, it restarts.

If the Memory Maximum Threshold limit is reached, all current operations abort. All client calls will be terminated. Once a call terminates, the corresponding document will be closed.

In short

When the server process memory is above the Memory Upper Threshold, the only operation allowed is saving documents.

When the server process memory is above the Memory Maximum Threshold, all operations stop and fail.

Screenshots

Content:

• BOBJ Metrics: cumulative Web Intelligence usage of 430 GB, with recommendations on Web Intelligence servers per host
• BOBJ Metrics: individual Web Intelligence usage reaching 74 GB
• OS-level monitoring: cumulative Web Intelligence allocation of 450 GB

Here is a production system with 800 sessions. Cumulative Web Intelligence memory usage reaches 430 GB, with individual Web Intelligence processes using up to 52 GB.

You can see multiple Web Intelligence servers per host. SAP generally recommends one Web Intelligence server per host, or two for failover if the system has only a few Web Intelligence nodes. Although we observed during VTOs that in some cases multiple Web Intelligence servers performed better, for simplicity and to reduce communication overhead I generally recommend fewer servers unless there is a clear reason to do otherwise.

Here is a VTO run with 1,400 sessions, with a Web Intelligence process consuming up to 74 GB.

OS-level monitoring shows that, on a system with only 46 active sessions, Web Intelligence on two nodes cumulatively consumes about 450 GB.

However, I do not believe this reflects real memory usage – it is more likely allocated memory. This is mainly due to the customer running 10 Web Intelligence servers per host, with Memory Analysis switched off and 1 TB of RAM available per processing node.