BiWhy - APS Split and Memory Configuration

[ilya_ulyanov]

Disclaimer:
BiWhy is available again for FREE to everyone on biwhy.net

Rule of thumb for APS Split – Less is More. Split only when you know why.

Bad Reason to Split APS

APS split is often assumed to improve performance. That assumption is wrong. It might be slightly faster due to more efficient garbage collection or reduced log contention, but under normal conditions the difference is negligible.

Good Reasons to Split APS

1. Troubleshooting – When increasing log level, you do not want logs from every APS service. You want logs from the one you are investigating.
2. Functional separation – For example:
• APS.DSLBridge on Processing nodes
• APS.Core on Intelligence (CMS) nodes
3. Separate troublemakers from workhorses – for example:
• APS.Search – historically problematic.
• APS.Monitoring – generates a lot of log noise.
• APS.DSLBridge – a primary workload engine; keep it isolated from noise and instability.
4. Proactive management – If APS.Core (hosting multiple services) is CPU- or memory-heavy, splitting helps identify which service is responsible.

Xmx

1. Few smaller identical processes (on the same host) vs one large process
   I frequently see multiple small identical APS processes, though in most cases one larger process is more efficient – especially in tight memory environments.
   Example (simplified, but based on real observations):
   You have 10 APS.DSLBridge instances with Xmx = 10GB each. Under current load, each uses 8GB. A new large report requires an additional 2+GB.
   Result:

• One APS hits 10GB – it starts struggling, response times spike, and users feel it immediately.
• Eventually it is killed.
• Load shifts to the remaining nine while the tenth process is restarting.
• Cascading failures begin.

   Now compare that with: One APS with Xmx = 100GB. Usage goes from 80GB to 82GB. No drama.

   Also, one process means one cache. This results in better cache efficiency and lower overall memory overhead.

1. Xmx as a capacity planning variable
   Sum of Xmx across host processes is usually the first or second largest memory allocation factor (after WIReportServer).
   Xmx sum is a theoretical limit – but it must be included in your production memory capacity planning calculation.

1. With high Xmx, consider disabling automatic memory dumps
   With very large Xmx values:

• Heap dumps take a long time to generate.
• They consume additional RAM during creation, while RAM is already a scarce resource.
• They quickly fill disk space.
• You need a very powerful machine to analyze them, and even then it will be slow.

If you need dumps for troubleshooting, consider creating a temporary process with smaller Xmx.

Xms

Avoid setting a high Xms.

It reserves memory from the OS that may never be needed. We are not in the business of spacecraft navigation – a few milliseconds of heap expansion delay do not matter.

I would say do not set Xms at all, but the JVM may set it automatically as a percentage of Xmx. It depends on the JVM implementation, so maybe stick with small or default values.

Real example:

There was a full escalation caused by Xms. Someone set Xms = Xmx = 6GB for Crystal Job Child Processes. When jobs kicked in, the host ran out of memory, the system slowed to a crawl, and errors were thrown everywhere. It took time to find the root cause – and those child processes did not even need that memory.

Base Split Recommendation

Minimalist approach:

• APS.Search (Intelligence Node) – Historically problematic.
  • If unused: clean indexes. You may find a large index file or multiple delta indexes. There is no explicit way to merge them, so use the following approach: Stop Indexing, run Full Content / Rebuild Index with unchecked documents and instances, restart APS, then Start Indexing. If there is a large number of delta indexes, they will be removed asynchronously even after APS.Search activity goes to zero.
  • If used (it is rare): follow best practices, SAP Note 2241459 – Platform Search Application in BI 4.x [LEAD KBA].
• APS.Monitoring (Intelligence Node) – Isolate, as it tends to generate a lot of log noise.
• APS.DSLBridge (Processing Nodes) – Treat as a primary workload engine if used.
• APS.LCM – Optional isolation if you run heavy transports. If transports run only off-hours, its Xmx may be excluded from peak host memory calculations.
• APS.Core – Everything else, unless you have a clear reason for further splitting.

Screenshots:

APS.Search Footprint

System has 238k objects in CMS DB.

Among them, 149k are delta search files consuming 14GB.

Xms = Xmx = 16GB

Sum of Xmx Nearly Double Available RAM

Total Xmx on host = 96GB.

Available RAM = 59GB.

Job Child Xmx Configuration – 8GB × 15

Xmx for job child processes = 8GB.

Job limit = 15 (limit was reached during observation).

Theoretical maximum consumption = 120GB.

Host RAM = 50GB.

Job Memory Configuration – 12GB × 50 = 600GB ! (Theoretical Maximum for Jobs Only)

Xmx for job child processes = 12GB.

WEBI job limit = 50.

Theoretical maximum consumption (jobs only) = 600GB.

Observed peak concurrent jobs = 28.

Host RAM = 64GB.

Total Xms across processes = 5GB.

APS Count and Inconsistent Xms Configuration

Is the number of APS instances excessive? Possibly.
If stable, it may be acceptable.

However:

• Why is Xms = 10GB?
• Why is Xms different across identical process types?