How to fix OOMKilled restart issue with SAP Commerce Cloud

[YannickRobin]

Problem Statement

In Dynatrace, you may see regular container restarts associated to the error message "The container platform was OOMKilled". 

We have noticed this is issue may happen with any JVM service including Solr.

Please note the error is at the container level and it should not be confused with Out-of-Memory error due to JVM heap saturation.

It can be confirmed by observing the container memory. We see that at the time of the restart, the memory usage of the container is very close to the container limit. The small gap is due to the container overhead.

Analysis

For deep-dive analysis of the root cause, it require to enable native memory tracking on the JVM.

ccv2.additional.catalina.opts=-XX:NativeMemoryTracking=detail -Xlog:nmt=trace,thread*=trace:file=VMlog.log:time,level,pid,tid,tags

Then you can raise a ticket to the Support to collect regularly data using jcmd utility with the command jcmd 1 VM.native_memory.

Analysing VMLog files, you will be able to calculate the total Native Memory Committed and understand which area of the non-heap memory keeps increasing until saturation.

In most cases, we have identified that the issue is not due to the non-heap memory growing but because of slow increase of memory fragmentation until RSS memory reaches the container limit minus the container overhead.

A common pattern for the memory fragmentation increase seems to be the high number of thread creations/terminations. The high Thread Stack allocation and deallocation throughput creates many small chunks of free memory over the time and increase RSS memory fragmentation.

Set Malloc Arena Max to reduce memory fragmentation

The GNU C library's (glibc's) malloc library contains a handful of functions that manage allocated memory in the application's address space.

This malloc is a "heap" style malloc, which means that chunks of various sizes exist within a larger region of memory (a "heap"). glibc's malloc allows for multiple heaps, each of which grows within its address space.

An arena is a structure that is shared among one or more threads which contains references to one or more heaps, as well as linked lists of chunks within those heaps which are "free". Threads assigned to each arena will allocate memory from that arena's free lists.

The workaround to reduce the memory fragmentation is to reduce the number of Malloc Arena.

Our benchmark with vanilla SAP Commerce Cloud Storefront stressed at very high load shows that the performance is not impacted by reducing the size.

Below is the outcome for 10 cores (default 80 malloc arenas).

On 64-bit machine, the default is 8 arenas per core assigned to the service and we suggest reducing to 2 arenas per core.

For this, we need to set MALLOC_ARENA_MAX as a new environment variable for the container causing the issue in CCV2 using SAP Commerce Cloud Portal.

Please note this is not setting a JVM parameter but setting a new environment variable so for this change and it requires to customize the Catalina startup script.

Below is the procedure:

• Override /opt/tomcat/bin/catalina.sh by configuring the service properties below

ccv2.file.override.70.content=<BASE64_CONTENT_CUSTOM_CATALINA>
ccv2.file.override.70.path=/opt/tomcat/bin/catalina.sh
ccv2.file.override.71.content=<BASE64_CONTENT_OF_ORIGINAL_FILE>
ccv2.file.override.71.path=/opt/tomcat/bin/catalina-original.sh

<BASE64_CONTENT_CUSTOM_CATALINA> is the content below encoded to Base64 format.

#!/usr/bin/env bash
set -e
echo "Set MALLOC ARENA MAX to 20"
export MALLOC_ARENA_MAX=20
chown hybris /opt/tomcat/bin/catalina-original.sh
chmod +x /opt/tomcat/bin/catalina-original.sh
exec /opt/tomcat/bin/catalina-original.sh run

For 64-bit machine, default value of MALLOC_ARENA_MAX is 8x the number of cores. 

To reduce memory defragmentation, we suggest setting 2x the number of cores for Service pod (ex: number of cores for Storefront service). Above script is setting MALLOC_ARENA_MAX value as 20 for a 10 cores pod.

<BASE64_CONTENT_OF_ORIGINAL_CATALINA_FILE> is the original /opt/tomcat/bin/catalina.sh encoded to Base64 format.

Note that in case of Tomcat upgrade, you need to encode again this value. For this, you can execute the following script in HAC after upgrade to check that the value did not change:

import org.apache.commons.io.FileUtils;
String contents = FileUtils.readFileToString(new File("/opt/tomcat/bin/catalina.sh"), "UTF-8")

Below is an example of properties with catalina-original.sh of SAP Commerce 2211.27 (Tomcat 9.0.86).

ccv2.file.override.70.content=IyEvdXNyL2Jpbi9lbnYgYmFzaApzZXQgLWUKZWNobyAiU2V0IE1BTExPQyBBUkVOQSBNQVggdG8gMjAiCmV4cG9ydCBNQUxMT0NfQVJFTkFfTUFYPTIwCmNob3duIGh5YnJpcyAvb3B0L3RvbWNhdC9iaW4vY2F0YWxpbmEtb3JpZ2luYWwuc2gKY2htb2QgK3ggL29wdC90b21jYXQvYmluL2NhdGFsaW5hLW9yaWdpbmFsLnNoCmV4ZWMgL29wdC90b21jYXQvYmluL2NhdGFsaW5hLW9yaWdpbmFsLnNoIHJ1bg==
ccv2.file.override.70.path=/opt/tomcat/bin/catalina.sh
ccv2.file.override.71.content=IyEvYmluL3NoCgojIExpY2Vuc2VkIHRvIHRoZSBBcGFjaGUgU29mdHdhcmUgRm91bmRhdGlvbiAoQVNGKSB1bmRlciBvbmUgb3IgbW9yZQojIGNvbnRyaWJ1dG9yIGxpY2Vuc2UgYWdyZWVtZW50cy4gIFNlZSB0aGUgTk9USUNFIGZpbGUgZGlzdHJpYnV0ZWQgd2l0aAojIHRoaXMgd29yayBmb3IgYWRkaXRpb25hbCBpbmZvcm1hdGlvbiByZWdhcmRpbmcgY29weXJpZ2h0IG93bmVyc2hpcC4KIyBUaGUgQVNGIGxpY2Vuc2VzIHRoaXMgZmlsZSB0byBZb3UgdW5kZXIgdGhlIEFwYWNoZSBMaWNlbnNlLCBWZXJzaW9uIDIuMAojICh0aGUgIkxpY2Vuc2UiKTsgeW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoCiMgdGhlIExpY2Vuc2UuICBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKIwojICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKIwojIFVubGVzcyByZXF1aXJlZCBieSBhcHBsaWNhYmxlIGxhdyBvciBhZ3JlZWQgdG8gaW4gd3JpdGluZywgc29mdHdhcmUKIyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAojIFdJVEhPVVQgV0FSUkFOVElFUyBPUiBDT05ESVRJT05TIE9GIEFOWSBLSU5ELCBlaXRoZXIgZXhwcmVzcyBvciBpbXBsaWVkLgojIFNlZSB0aGUgTGljZW5zZSBmb3IgdGhlIHNwZWNpZmljIGxhbmd1YWdlIGdvdmVybmluZyBwZXJtaXNzaW9ucyBhbmQKIyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCiMgLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0KIyBDb250cm9sIFNjcmlwdCBmb3IgdGhlIENBVEFMSU5BIFNlcnZlcgojCiMgRm9yIHN1cHBvcnRlZCBjb21tYW5kcyBjYWxsICJjYXRhbGluYS5zaCBoZWxwIiBvciBzZWUgdGhlIHVzYWdlIHNlY3Rpb24gYXQKIyB0aGUgZW5kIG9mIHRoaXMgZmlsZS4KIwojIEVudmlyb25tZW50IFZhcmlhYmxlIFByZXJlcXVpc2l0ZXMKIwojICAgRG8gbm90IHNldCB0aGUgdmFyaWFibGVzIGluIHRoaXMgc2NyaXB0LiBJbnN0ZWFkIHB1dCB0aGVtIGludG8gYSBzY3JpcHQKIyAgIHNldGVudi5zaCBpbiBDQVRBTElOQV9CQVNFL2JpbiB0byBrZWVwIHlvdXIgY3VzdG9taXphdGlvbnMgc2VwYXJhdGUuCiMKIyAgIENBVEFMSU5BX0hPTUUgICBNYXkgcG9pbnQgYXQgeW91ciBDYXRhbGluYSAiYnVpbGQiIGRpcmVjdG9yeS4KIwojICAgQ0FUQUxJTkFfQkFTRSAgIChPcHRpb25hbCkgQmFzZSBkaXJlY3RvcnkgZm9yIHJlc29sdmluZyBkeW5hbWljIHBvcnRpb25zCiMgICAgICAgICAgICAgICAgICAgb2YgYSBDYXRhbGluYSBpbnN0YWxsYXRpb24uICBJZiBub3QgcHJlc2VudCwgcmVzb2x2ZXMgdG8KIyAgICAgICAgICAgICAgICAgICB0aGUgc2FtZSBkaXJlY3RvcnkgdGhhdCBDQVRBTElOQV9IT01FIHBvaW50cyB0by4KIwojICAgQ0FUQUxJTkFfT1VUICAgIChPcHRpb25hbCkgRnVsbCBwYXRoIHRvIGEgZmlsZSB3aGVyZSBzdGRvdXQgYW5kIHN0ZGVycgojICAgICAgICAgICAgICAgICAgIHdpbGwgYmUgcmVkaXJlY3RlZC4KIyAgICAgICAgICAgICAgICAgICBEZWZhdWx0IGlzICRDQVRBTElOQV9CQVNFL2xvZ3MvY2F0YWxpbmEub3V0CiMKIyAgIENBVEFMSU5BX09VVF9DTUQgKE9wdGlvbmFsKSBDb21tYW5kIHdoaWNoIHdpbGwgYmUgZXhlY3V0ZWQgYW5kIHJlY2VpdmUKIyAgICAgICAgICAgICAgICAgICBhcyBpdHMgc3RkaW4gdGhlIHN0ZG91dCBhbmQgc3RkZXJyIGZyb20gdGhlIFRvbWNhdCBqYXZhCiMgICAgICAgICAgICAgICAgICAgcHJvY2Vzcy4gSWYgQ0FUQUxJTkFfT1VUX0NNRCBpcyBzZXQsIHRoZSB2YWx1ZSBvZgojICAgICAgICAgICAgICAgICAgIENBVEFMSU5BX09VVCB3aWxsIGJlIHVzZWQgYXMgYSBuYW1lZCBwaXBlLgojICAgICAgICAgICAgICAgICAgIE5vIGRlZmF1bHQuCiMgICAgICAgICAgICAgICAgICAgRXhhbXBsZSAoYWxsIG9uZSBsaW5lKQojICAgICAgICAgICAgICAgICAgIENBVEFMSU5BX09VVF9DTUQ9Ii91c3IvYmluL3JvdGF0ZWxvZ3MgLWYgJENBVEFMSU5BX0JBU0UvbG9ncy9jYXRhbGluYS5vdXQuJVktJW0tJWQubG9nIDg2NDAwIgojCiMgICBDQVRBTElOQV9PUFRTICAgKE9wdGlvbmFsKSBKYXZhIHJ1bnRpbWUgb3B0aW9ucyB1c2VkIHdoZW4gdGhlICJzdGFydCIsCiMgICAgICAgICAgICAgICAgICAgInJ1biIgb3IgImRlYnVnIiBjb21tYW5kIGlzIGV4ZWN1dGVkLgojICAgICAgICAgICAgICAgICAgIEluY2x1ZGUgaGVyZSBhbmQgbm90IGluIEpBVkFfT1BUUyBhbGwgb3B0aW9ucywgdGhhdCBzaG91bGQKIyAgICAgICAgICAgICAgICAgICBvbmx5IGJlIHVzZWQgYnkgVG9tY2F0IGl0c2VsZiwgbm90IGJ5IHRoZSBzdG9wIHByb2Nlc3MsCiMgICAgICAgICAgICAgICAgICAgdGhlIHZlcnNpb24gY29tbWFuZCBldGMuCiMgICAgICAgICAgICAgICAgICAgRXhhbXBsZXMgYXJlIGhlYXAgc2l6ZSwgR0MgbG9nZ2luZywgSk1YIHBvcnRzIGV0Yy4KIwojICAgQ0FUQUxJTkFfVE1QRElSIChPcHRpb25hbCkgRGlyZWN0b3J5IHBhdGggbG9jYXRpb24gb2YgdGVtcG9yYXJ5IGRpcmVjdG9yeQojICAgICAgICAgICAgICAgICAgIHRoZSBKVk0gc2hvdWxkIHVzZSAoamF2YS5pby50bXBkaXIpLiAgRGVmYXVsdHMgdG8KIyAgICAgICAgICAgICAgICAgICAkQ0FUQUxJTkFfQkFTRS90ZW1wLgojCiMgICBKQVZBX0hPTUUgICAgICAgTXVzdCBwb2ludCBhdCB5b3VyIEphdmEgRGV2ZWxvcG1lbnQgS2l0IGluc3RhbGxhdGlvbi4KIyAgICAgICAgICAgICAgICAgICBSZXF1aXJlZCB0byBydW4gdGhlIHdpdGggdGhlICJkZWJ1ZyIgYXJndW1lbnQuCiMKIyAgIEpSRV9IT01FICAgICAgICBNdXN0IHBvaW50IGF0IHlvdXIgSmF2YSBSdW50aW1lIGluc3RhbGxhdGlvbi4KIyAgICAgICAgICAgICAgICAgICBEZWZhdWx0cyB0byBKQVZBX0hPTUUgaWYgZW1wdHkuIElmIEpSRV9IT01FIGFuZCBKQVZBX0hPTUUKIyAgICAgICAgICAgICAgICAgICBhcmUgYm90aCBzZXQsIEpSRV9IT01FIGlzIHVzZWQuCiMKIyAgIEpBVkFfT1BUUyAgICAgICAoT3B0aW9uYWwpIEphdmEgcnVudGltZSBvcHRpb25zIHVzZWQgd2hlbiBhbnkgY29tbWFuZAojICAgICAgICAgICAgICAgICAgIGlzIGV4ZWN1dGVkLgojICAgICAgICAgICAgICAgICAgIEluY2x1ZGUgaGVyZSBhbmQgbm90IGluIENBVEFMSU5BX09QVFMgYWxsIG9wdGlvbnMsIHRoYXQKIyAgICAgICAgICAgICAgICAgICBzaG91bGQgYmUgdXNlZCBieSBUb21jYXQgYW5kIGFsc28gYnkgdGhlIHN0b3AgcHJvY2VzcywKIyAgICAgICAgICAgICAgICAgICB0aGUgdmVyc2lvbiBjb21tYW5kIGV0Yy4KIyAgICAgICAgICAgICAgICAgICBNb3N0IG9wdGlvbnMgc2hvdWxkIGdvIGludG8gQ0FUQUxJTkFfT1BUUy4KIwojICAgSkFWQV9FTkRPUlNFRF9ESVJTIChPcHRpb25hbCkgTGlzdHMgb2Ygb2YgY29sb24gc2VwYXJhdGVkIGRpcmVjdG9yaWVzCiMgICAgICAgICAgICAgICAgICAgY29udGFpbmluZyBzb21lIGphcnMgaW4gb3JkZXIgdG8gYWxsb3cgcmVwbGFjZW1lbnQgb2YgQVBJcwojICAgICAgICAgICAgICAgICAgIGNyZWF0ZWQgb3V0c2lkZSBvZiB0aGUgSkNQIChpLmUuIERPTSBhbmQgU0FYIGZyb20gVzNDKS4KIyAgICAgICAgICAgICAgICAgICBJdCBjYW4gYWxzbyBiZSB1c2VkIHRvIHVwZGF0ZSB0aGUgWE1MIHBhcnNlciBpbXBsZW1lbnRhdGlvbi4KIyAgICAgICAgICAgICAgICAgICBUaGlzIGlzIG9ubHkgc3VwcG9ydGVkIGZvciBKYXZhIDw9IDguCiMgICAgICAgICAgICAgICAgICAgRGVmYXVsdHMgdG8gJENBVEFMSU5BX0hPTUUvZW5kb3JzZWQuCiMKIyAgIEpQREFfVFJBTlNQT1JUICAoT3B0aW9uYWwpIEpQREEgdHJhbnNwb3J0IHVzZWQgd2hlbiB0aGUgImpwZGEgc3RhcnQiCiMgICAgICAgICAgICAgICAgICAgY29tbWFuZCBpcyBleGVjdXRlZC4gVGhlIGRlZmF1bHQgaXMgImR0X3NvY2tldCIuCiMKIyAgIEpQREFfQUREUkVTUyAgICAoT3B0aW9uYWwpIEphdmEgcnVudGltZSBvcHRpb25zIHVzZWQgd2hlbiB0aGUgImpwZGEgc3RhcnQiCiMgICAgICAgICAgICAgICAgICAgY29tbWFuZCBpcyBleGVjdXRlZC4gVGhlIGRlZmF1bHQgaXMgbG9jYWxob3N0OjgwMDAuCiMKIyAgIEpQREFfU1VTUEVORCAgICAoT3B0aW9uYWwpIEphdmEgcnVudGltZSBvcHRpb25zIHVzZWQgd2hlbiB0aGUgImpwZGEgc3RhcnQiCiMgICAgICAgICAgICAgICAgICAgY29tbWFuZCBpcyBleGVjdXRlZC4gU3BlY2lmaWVzIHdoZXRoZXIgSlZNIHNob3VsZCBzdXNwZW5kCiMgICAgICAgICAgICAgICAgICAgZXhlY3V0aW9uIGltbWVkaWF0ZWx5IGFmdGVyIHN0YXJ0dXAuIERlZmF1bHQgaXMgIm4iLgojCiMgICBKUERBX09QVFMgICAgICAgKE9wdGlvbmFsKSBKYXZhIHJ1bnRpbWUgb3B0aW9ucyB1c2VkIHdoZW4gdGhlICJqcGRhIHN0YXJ0IgojICAgICAgICAgICAgICAgICAgIGNvbW1hbmQgaXMgZXhlY3V0ZWQuIElmIHVzZWQsIEpQREFfVFJBTlNQT1JULCBKUERBX0FERFJFU1MsCiMgICAgICAgICAgICAgICAgICAgYW5kIEpQREFfU1VTUEVORCBhcmUgaWdub3JlZC4gVGh1cywgYWxsIHJlcXVpcmVkIGpwZGEKIyAgICAgICAgICAgICAgICAgICBvcHRpb25zIE1VU1QgYmUgc3BlY2lmaWVkLiBUaGUgZGVmYXVsdCBpczoKIwojICAgICAgICAgICAgICAgICAgIC1hZ2VudGxpYjpqZHdwPXRyYW5zcG9ydD0kSlBEQV9UUkFOU1BPUlQsCiMgICAgICAgICAgICAgICAgICAgICAgIGFkZHJlc3M9JEpQREFfQUREUkVTUyxzZXJ2ZXI9eSxzdXNwZW5kPSRKUERBX1NVU1BFTkQKIwojICAgSlNTRV9PUFRTICAgICAgIChPcHRpb25hbCkgSmF2YSBydW50aW1lIG9wdGlvbnMgdXNlZCB0byBjb250cm9sIHRoZSBUTFMKIyAgICAgICAgICAgICAgICAgICBpbXBsZW1lbnRhdGlvbiB3aGVuIEpTU0UgaXMgdXNlZC4gRGVmYXVsdCBpczoKIyAgICAgICAgICAgICAgICAgICAiLURqZGsudGxzLmVwaGVtZXJhbERIS2V5U2l6ZT0yMDQ4IgojCiMgICBDQVRBTElOQV9QSUQgICAgKE9wdGlvbmFsKSBQYXRoIG9mIHRoZSBmaWxlIHdoaWNoIHNob3VsZCBjb250YWlucyB0aGUgcGlkCiMgICAgICAgICAgICAgICAgICAgb2YgdGhlIGNhdGFsaW5hIHN0YXJ0dXAgamF2YSBwcm9jZXNzLCB3aGVuIHN0YXJ0IChmb3JrKSBpcwojICAgICAgICAgICAgICAgICAgIHVzZWQKIwojICAgQ0FUQUxJTkFfTE9HR0lOR19DT05GSUcgKE9wdGlvbmFsKSBPdmVycmlkZSBUb21jYXQncyBsb2dnaW5nIGNvbmZpZyBmaWxlCiMgICAgICAgICAgICAgICAgICAgRXhhbXBsZSAoYWxsIG9uZSBsaW5lKQojICAgICAgICAgICAgICAgICAgIENBVEFMSU5BX0xPR0dJTkdfQ09ORklHPSItRGphdmEudXRpbC5sb2dnaW5nLmNvbmZpZy5maWxlPSRDQVRBTElOQV9CQVNFL2NvbmYvbG9nZ2luZy5wcm9wZXJ0aWVzIgojCiMgICBMT0dHSU5HX0NPTkZJRyAgRGVwcmVjYXRlZAojICAgICAgICAgICAgICAgICAgIFVzZSBDQVRBTElOQV9MT0dHSU5HX0NPTkZJRwojICAgICAgICAgICAgICAgICAgIFRoaXMgaXMgb25seSB1c2VkIGlmIENBVEFMSU5BX0xPR0dJTkdfQ09ORklHIGlzIG5vdCBzZXQKIyAgICAgICAgICAgICAgICAgICBhbmQgTE9HR0lOR19DT05GSUcgc3RhcnRzIHdpdGggIi1ELi4uIgojCiMgICBMT0dHSU5HX01BTkFHRVIgKE9wdGlvbmFsKSBPdmVycmlkZSBUb21jYXQncyBsb2dnaW5nIG1hbmFnZXIKIyAgICAgICAgICAgICAgICAgICBFeGFtcGxlIChhbGwgb25lIGxpbmUpCiMgICAgICAgICAgICAgICAgICAgTE9HR0lOR19NQU5BR0VSPSItRGphdmEudXRpbC5sb2dnaW5nLm1hbmFnZXI9b3JnLmFwYWNoZS5qdWxpLkNsYXNzTG9hZGVyTG9nTWFuYWdlciIKIwojICAgVU1BU0sgICAgICAgICAgIChPcHRpb25hbCkgT3ZlcnJpZGUgVG9tY2F0J3MgZGVmYXVsdCBVTUFTSyBvZiAwMDI3CiMKIyAgIFVTRV9OT0hVUCAgICAgICAoT3B0aW9uYWwpIElmIHNldCB0byB0aGUgc3RyaW5nIHRydWUgdGhlIHN0YXJ0IGNvbW1hbmQgd2lsbAojICAgICAgICAgICAgICAgICAgIHVzZSBub2h1cCBzbyB0aGF0IHRoZSBUb21jYXQgcHJvY2VzcyB3aWxsIGlnbm9yZSBhbnkgaGFuZ3VwCiMgICAgICAgICAgICAgICAgICAgc2lnbmFscy4gRGVmYXVsdCBpcyAiZmFsc2UiIHVubGVzcyBydW5uaW5nIG9uIEhQLVVYIGluIHdoaWNoCiMgICAgICAgICAgICAgICAgICAgY2FzZSB0aGUgZGVmYXVsdCBpcyAidHJ1ZSIKIyAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQoKIyBPUyBzcGVjaWZpYyBzdXBwb3J0LiAgJHZhciBfbXVzdF8gYmUgc2V0IHRvIGVpdGhlciB0cnVlIG9yIGZhbHNlLgpjeWd3aW49ZmFsc2UKZGFyd2luPWZhbHNlCm9zNDAwPWZhbHNlCmhwdXg9ZmFsc2UKY2FzZSAiYHVuYW1lYCIgaW4KQ1lHV0lOKikgY3lnd2luPXRydWU7OwpEYXJ3aW4qKSBkYXJ3aW49dHJ1ZTs7Ck9TNDAwKikgb3M0MDA9dHJ1ZTs7CkhQLVVYKikgaHB1eD10cnVlOzsKZXNhYwoKIyByZXNvbHZlIGxpbmtzIC0gJDAgbWF5IGJlIGEgc29mdGxpbmsKUFJHPSIkMCIKCndoaWxlIFsgLWggIiRQUkciIF07IGRvCiAgbHM9YGxzIC1sZCAiJFBSRyJgCiAgbGluaz1gZXhwciAiJGxzIiA6ICcuKi0+IFwoLipcKSQnYAogIGlmIGV4cHIgIiRsaW5rIiA6ICcvLionID4gL2Rldi9udWxsOyB0aGVuCiAgICBQUkc9IiRsaW5rIgogIGVsc2UKICAgIFBSRz1gZGlybmFtZSAiJFBSRyJgLyIkbGluayIKICBmaQpkb25lCgojIEdldCBzdGFuZGFyZCBlbnZpcm9ubWVudCB2YXJpYWJsZXMKUFJHRElSPWBkaXJuYW1lICIkUFJHImAKCiMgT25seSBzZXQgQ0FUQUxJTkFfSE9NRSBpZiBub3QgYWxyZWFkeSBzZXQKWyAteiAiJENBVEFMSU5BX0hPTUUiIF0gJiYgQ0FUQUxJTkFfSE9NRT1gY2QgIiRQUkdESVIvLi4iID4vZGV2L251bGw7IHB3ZGAKCiMgQ29weSBDQVRBTElOQV9CQVNFIGZyb20gQ0FUQUxJTkFfSE9NRSBpZiBub3QgYWxyZWFkeSBzZXQKWyAteiAiJENBVEFMSU5BX0JBU0UiIF0gJiYgQ0FUQUxJTkFfQkFTRT0iJENBVEFMSU5BX0hPTUUiCgojIEVuc3VyZSB0aGF0IGFueSB1c2VyIGRlZmluZWQgQ0xBU1NQQVRIIHZhcmlhYmxlcyBhcmUgbm90IHVzZWQgb24gc3RhcnR1cCwKIyBidXQgYWxsb3cgdGhlbSB0byBiZSBzcGVjaWZpZWQgaW4gc2V0ZW52LnNoLCBpbiByYXJlIGNhc2Ugd2hlbiBpdCBpcyBuZWVkZWQuCkNMQVNTUEFUSD0KCmlmIFsgLXIgIiRDQVRBTElOQV9CQVNFL2Jpbi9zZXRlbnYuc2giIF07IHRoZW4KICAuICIkQ0FUQUxJTkFfQkFTRS9iaW4vc2V0ZW52LnNoIgplbGlmIFsgLXIgIiRDQVRBTElOQV9IT01FL2Jpbi9zZXRlbnYuc2giIF07IHRoZW4KICAuICIkQ0FUQUxJTkFfSE9NRS9iaW4vc2V0ZW52LnNoIgpmaQoKIyBGb3IgQ3lnd2luLCBlbnN1cmUgcGF0aHMgYXJlIGluIFVOSVggZm9ybWF0IGJlZm9yZSBhbnl0aGluZyBpcyB0b3VjaGVkCmlmICRjeWd3aW47IHRoZW4KICBbIC1uICIkSkFWQV9IT01FIiBdICYmIEpBVkFfSE9NRT1gY3lncGF0aCAtLXVuaXggIiRKQVZBX0hPTUUiYAogIFsgLW4gIiRKUkVfSE9NRSIgXSAmJiBKUkVfSE9NRT1gY3lncGF0aCAtLXVuaXggIiRKUkVfSE9NRSJgCiAgWyAtbiAiJENBVEFMSU5BX0hPTUUiIF0gJiYgQ0FUQUxJTkFfSE9NRT1gY3lncGF0aCAtLXVuaXggIiRDQVRBTElOQV9IT01FImAKICBbIC1uICIkQ0FUQUxJTkFfQkFTRSIgXSAmJiBDQVRBTElOQV9CQVNFPWBjeWdwYXRoIC0tdW5peCAiJENBVEFMSU5BX0JBU0UiYAogIFsgLW4gIiRDTEFTU1BBVEgiIF0gJiYgQ0xBU1NQQVRIPWBjeWdwYXRoIC0tcGF0aCAtLXVuaXggIiRDTEFTU1BBVEgiYApmaQoKIyBFbnN1cmUgdGhhdCBuZWl0aGVyIENBVEFMSU5BX0hPTUUgbm9yIENBVEFMSU5BX0JBU0UgY29udGFpbnMgYSBjb2xvbgojIGFzIHRoaXMgaXMgdXNlZCBhcyB0aGUgc2VwYXJhdG9yIGluIHRoZSBjbGFzc3BhdGggYW5kIEphdmEgcHJvdmlkZXMgbm8KIyBtZWNoYW5pc20gZm9yIGVzY2FwaW5nIGlmIHRoZSBzYW1lIGNoYXJhY3RlciBhcHBlYXJzIGluIHRoZSBwYXRoLgpjYXNlICRDQVRBTElOQV9IT01FIGluCiAgKjoqKSBlY2hvICJVc2luZyBDQVRBTElOQV9IT01FOiAgICRDQVRBTElOQV9IT01FIjsKICAgICAgIGVjaG8gIlVuYWJsZSB0byBzdGFydCBhcyBDQVRBTElOQV9IT01FIGNvbnRhaW5zIGEgY29sb24gKDopIGNoYXJhY3RlciI7CiAgICAgICBleGl0IDE7CmVzYWMKY2FzZSAkQ0FUQUxJTkFfQkFTRSBpbgogICo6KikgZWNobyAiVXNpbmcgQ0FUQUxJTkFfQkFTRTogICAkQ0FUQUxJTkFfQkFTRSI7CiAgICAgICBlY2hvICJVbmFibGUgdG8gc3RhcnQgYXMgQ0FUQUxJTkFfQkFTRSBjb250YWlucyBhIGNvbG9uICg6KSBjaGFyYWN0ZXIiOwogICAgICAgZXhpdCAxOwplc2FjCgojIEZvciBPUzQwMAppZiAkb3M0MDA7IHRoZW4KICAjIFNldCBqb2IgcHJpb3JpdHkgdG8gc3RhbmRhcmQgZm9yIGludGVyYWN0aXZlIChpbnRlcmFjdGl2ZSAtIDYpIGJ5IHVzaW5nCiAgIyB0aGUgaW50ZXJhY3RpdmUgcHJpb3JpdHkgLSA2LCB0aGUgaGVscGVyIHRocmVhZHMgdGhhdCByZXNwb25kIHRvIHJlcXVlc3RzCiAgIyB3aWxsIGJlIHJ1bm5pbmcgYXQgdGhlIHNhbWUgcHJpb3JpdHkgYXMgaW50ZXJhY3RpdmUgam9icy4KICBDT01NQU5EPSdjaGdqb2Igam9iKCckSk9CTkFNRScpIHJ1bnB0eSg2KScKICBzeXN0ZW0gJENPTU1BTkQKCiAgIyBFbmFibGUgbXVsdGkgdGhyZWFkaW5nCiAgZXhwb3J0IFFJQk1fTVVMVElfVEhSRUFERUQ9WQpmaQoKIyBHZXQgc3RhbmRhcmQgSmF2YSBlbnZpcm9ubWVudCB2YXJpYWJsZXMKaWYgJG9zNDAwOyB0aGVuCiAgIyAtciB3aWxsIE9ubHkgd29yayBvbiB0aGUgb3M0MDAgaWYgdGhlIGZpbGVzIGFyZToKICAjIDEuIG93bmVkIGJ5IHRoZSB1c2VyCiAgIyAyLiBvd25lZCBieSB0aGUgUFJJTUFSWSBncm91cCBvZiB0aGUgdXNlcgogICMgdGhpcyB3aWxsIG5vdCB3b3JrIGlmIHRoZSB1c2VyIGJlbG9uZ3MgaW4gc2Vjb25kYXJ5IGdyb3VwcwogIC4gIiRDQVRBTElOQV9IT01FIi9iaW4vc2V0Y2xhc3NwYXRoLnNoCmVsc2UKICBpZiBbIC1yICIkQ0FUQUxJTkFfSE9NRSIvYmluL3NldGNsYXNzcGF0aC5zaCBdOyB0aGVuCiAgICAuICIkQ0FUQUxJTkFfSE9NRSIvYmluL3NldGNsYXNzcGF0aC5zaAogIGVsc2UKICAgIGVjaG8gIkNhbm5vdCBmaW5kICRDQVRBTElOQV9IT01FL2Jpbi9zZXRjbGFzc3BhdGguc2giCiAgICBlY2hvICJUaGlzIGZpbGUgaXMgbmVlZGVkIHRvIHJ1biB0aGlzIHByb2dyYW0iCiAgICBleGl0IDEKICBmaQpmaQoKIyBBZGQgb24gZXh0cmEgamFyIGZpbGVzIHRvIENMQVNTUEFUSAppZiBbICEgLXogIiRDTEFTU1BBVEgiIF0gOyB0aGVuCiAgQ0xBU1NQQVRIPSIkQ0xBU1NQQVRIIjoKZmkKQ0xBU1NQQVRIPSIkQ0xBU1NQQVRIIiIkQ0FUQUxJTkFfSE9NRSIvYmluL2Jvb3RzdHJhcC5qYXIKCmlmIFsgLXogIiRDQVRBTElOQV9PVVQiIF0gOyB0aGVuCiAgQ0FUQUxJTkFfT1VUPSIkQ0FUQUxJTkFfQkFTRSIvbG9ncy9jYXRhbGluYS5vdXQKZmkKCmlmIFsgLXogIiRDQVRBTElOQV9UTVBESVIiIF0gOyB0aGVuCiAgIyBEZWZpbmUgdGhlIGphdmEuaW8udG1wZGlyIHRvIHVzZSBmb3IgQ2F0YWxpbmEKICBDQVRBTElOQV9UTVBESVI9IiRDQVRBTElOQV9CQVNFIi90ZW1wCmZpCgojIEFkZCB0b21jYXQtanVsaS5qYXIgdG8gY2xhc3NwYXRoCiMgdG9tY2F0LWp1bGkuamFyIGNhbiBiZSBvdmVyLXJpZGRlbiBwZXIgaW5zdGFuY2UKaWYgWyAtciAiJENBVEFMSU5BX0JBU0UvYmluL3RvbWNhdC1qdWxpLmphciIgXSA7IHRoZW4KICBDTEFTU1BBVEg9JENMQVNTUEFUSDokQ0FUQUxJTkFfQkFTRS9iaW4vdG9tY2F0LWp1bGkuamFyCmVsc2UKICBDTEFTU1BBVEg9JENMQVNTUEFUSDokQ0FUQUxJTkFfSE9NRS9iaW4vdG9tY2F0LWp1bGkuamFyCmZpCgojIEJ1Z3ppbGxhIDM3ODQ4OiBXaGVuIG5vIFRUWSBpcyBhdmFpbGFibGUsIGRvbid0IG91dHB1dCB0byBjb25zb2xlCmhhdmVfdHR5PTAKaWYgWyAtdCAwIF07IHRoZW4KICAgIGhhdmVfdHR5PTEKZmkKCiMgRm9yIEN5Z3dpbiwgc3dpdGNoIHBhdGhzIHRvIFdpbmRvd3MgZm9ybWF0IGJlZm9yZSBydW5uaW5nIGphdmEKaWYgJGN5Z3dpbjsgdGhlbgogIEpBVkFfSE9NRT1gY3lncGF0aCAtLWFic29sdXRlIC0td2luZG93cyAiJEpBVkFfSE9NRSJgCiAgSlJFX0hPTUU9YGN5Z3BhdGggLS1hYnNvbHV0ZSAtLXdpbmRvd3MgIiRKUkVfSE9NRSJgCiAgQ0FUQUxJTkFfSE9NRT1gY3lncGF0aCAtLWFic29sdXRlIC0td2luZG93cyAiJENBVEFMSU5BX0hPTUUiYAogIENBVEFMSU5BX0JBU0U9YGN5Z3BhdGggLS1hYnNvbHV0ZSAtLXdpbmRvd3MgIiRDQVRBTElOQV9CQVNFImAKICBDQVRBTElOQV9UTVBESVI9YGN5Z3BhdGggLS1hYnNvbHV0ZSAtLXdpbmRvd3MgIiRDQVRBTElOQV9UTVBESVIiYAogIENMQVNTUEFUSD1gY3lncGF0aCAtLXBhdGggLS13aW5kb3dzICIkQ0xBU1NQQVRIImAKICBbIC1uICIkSkFWQV9FTkRPUlNFRF9ESVJTIiBdICYmIEpBVkFfRU5ET1JTRURfRElSUz1gY3lncGF0aCAtLXBhdGggLS13aW5kb3dzICIkSkFWQV9FTkRPUlNFRF9ESVJTImAKZmkKCmlmIFsgLXogIiRKU1NFX09QVFMiIF0gOyB0aGVuCiAgSlNTRV9PUFRTPSItRGpkay50bHMuZXBoZW1lcmFsREhLZXlTaXplPTIwNDgiCmZpCkpBVkFfT1BUUz0iJEpBVkFfT1BUUyAkSlNTRV9PUFRTIgoKIyBSZWdpc3RlciBjdXN0b20gVVJMIGhhbmRsZXJzCiMgRG8gdGhpcyBoZXJlIHNvIGN1c3RvbSBVUkwgaGFuZGxlcyAoc3BlY2lmaWNhbGx5ICd3YXI6Li4uJykgY2FuIGJlIHVzZWQgaW4gdGhlIHNlY3VyaXR5IHBvbGljeQpKQVZBX09QVFM9IiRKQVZBX09QVFMgLURqYXZhLnByb3RvY29sLmhhbmRsZXIucGtncz1vcmcuYXBhY2hlLmNhdGFsaW5hLndlYnJlc291cmNlcyIKCiMgQ2hlY2sgZm9yIHRoZSBkZXByZWNhdGVkIExPR0dJTkdfQ09ORklHCiMgT25seSB1c2UgaXQgaWYgQ0FUQUxJTkFfTE9HR0lOR19DT05GSUcgaXMgbm90IHNldCBhbmQgTE9HR0lOR19DT05GSUcgc3RhcnRzIHdpdGggIi1ELi4uIgppZiBbIC16ICIkQ0FUQUxJTkFfTE9HR0lOR19DT05GSUciIF07IHRoZW4KICBjYXNlICRMT0dHSU5HX0NPTkZJRyBpbgogICAgLUQqKSBDQVRBTElOQV9MT0dHSU5HX0NPTkZJRz0iJExPR0dJTkdfQ09ORklHIgogIGVzYWMKZmkKCiMgU2V0IGp1bGkgTG9nTWFuYWdlciBjb25maWcgZmlsZSBpZiBpdCBpcyBwcmVzZW50IGFuZCBhbiBvdmVycmlkZSBoYXMgbm90IGJlZW4gaXNzdWVkCmlmIFsgLXogIiRDQVRBTElOQV9MT0dHSU5HX0NPTkZJRyIgXTsgdGhlbgogIGlmIFsgLXIgIiRDQVRBTElOQV9CQVNFIi9jb25mL2xvZ2dpbmcucHJvcGVydGllcyBdOyB0aGVuCiAgICBDQVRBTElOQV9MT0dHSU5HX0NPTkZJRz0iLURqYXZhLnV0aWwubG9nZ2luZy5jb25maWcuZmlsZT0kQ0FUQUxJTkFfQkFTRS9jb25mL2xvZ2dpbmcucHJvcGVydGllcyIKICBlbHNlCiAgICAjIEJ1Z3ppbGxhIDQ1NTg1CiAgICBDQVRBTElOQV9MT0dHSU5HX0NPTkZJRz0iLURub3AiCiAgZmkKZmkKCmlmIFsgLXogIiRMT0dHSU5HX01BTkFHRVIiIF07IHRoZW4KICBMT0dHSU5HX01BTkFHRVI9Ii1EamF2YS51dGlsLmxvZ2dpbmcubWFuYWdlcj1vcmcuYXBhY2hlLmp1bGkuQ2xhc3NMb2FkZXJMb2dNYW5hZ2VyIgpmaQoKIyBTZXQgVU1BU0sgdW5sZXNzIGl0IGhhcyBiZWVuIG92ZXJyaWRkZW4KaWYgWyAteiAiJFVNQVNLIiBdOyB0aGVuCiAgICBVTUFTSz0iMDAyNyIKZmkKdW1hc2sgJFVNQVNLCgojIEphdmEgOSBubyBsb25nZXIgc3VwcG9ydHMgdGhlIGphdmEuZW5kb3JzZWQuZGlycwojIHN5c3RlbSBwcm9wZXJ0eS4gT25seSB0cnkgdG8gdXNlIGl0IGlmCiMgSkFWQV9FTkRPUlNFRF9ESVJTIHdhcyBleHBsaWNpdGx5IHNldAojIG9yIENBVEFMSU5BX0hPTUUvZW5kb3JzZWQgZXhpc3RzLgpFTkRPUlNFRF9QUk9QPWlnbm9yZS5lbmRvcnNlZC5kaXJzCmlmIFsgLW4gIiRKQVZBX0VORE9SU0VEX0RJUlMiIF07IHRoZW4KICAgIEVORE9SU0VEX1BST1A9amF2YS5lbmRvcnNlZC5kaXJzCmZpCmlmIFsgLWQgIiRDQVRBTElOQV9IT01FL2VuZG9yc2VkIiBdOyB0aGVuCiAgICBFTkRPUlNFRF9QUk9QPWphdmEuZW5kb3JzZWQuZGlycwpmaQoKIyBNYWtlIHRoZSB1bWFzayBhdmFpbGFibGUgd2hlbiB1c2luZyB0aGUgb3JnLmFwYWNoZS5jYXRhbGluYS5zZWN1cml0eS5TZWN1cml0eUxpc3RlbmVyCkpBVkFfT1BUUz0iJEpBVkFfT1BUUyAtRG9yZy5hcGFjaGUuY2F0YWxpbmEuc2VjdXJpdHkuU2VjdXJpdHlMaXN0ZW5lci5VTUFTSz1gdW1hc2tgIgoKaWYgWyAteiAiJFVTRV9OT0hVUCIgXTsgdGhlbgogICAgaWYgJGhwdXg7IHRoZW4KICAgICAgICBVU0VfTk9IVVA9InRydWUiCiAgICBlbHNlCiAgICAgICAgVVNFX05PSFVQPSJmYWxzZSIKICAgIGZpCmZpCnVuc2V0IF9OT0hVUAppZiBbICIkVVNFX05PSFVQIiA9ICJ0cnVlIiBdOyB0aGVuCiAgICBfTk9IVVA9Im5vaHVwIgpmaQoKIyBBZGQgdGhlIEpBVkEgOSBzcGVjaWZpYyBzdGFydC11cCBwYXJhbWV0ZXJzIHJlcXVpcmVkIGJ5IFRvbWNhdApKREtfSkFWQV9PUFRJT05TPSIkSkRLX0pBVkFfT1BUSU9OUyAtLWFkZC1vcGVucz1qYXZhLmJhc2UvamF2YS5sYW5nPUFMTC1VTk5BTUVEIgpKREtfSkFWQV9PUFRJT05TPSIkSkRLX0pBVkFfT1BUSU9OUyAtLWFkZC1vcGVucz1qYXZhLmJhc2UvamF2YS5pbz1BTEwtVU5OQU1FRCIKSkRLX0pBVkFfT1BUSU9OUz0iJEpES19KQVZBX09QVElPTlMgLS1hZGQtb3BlbnM9amF2YS5iYXNlL2phdmEudXRpbD1BTEwtVU5OQU1FRCIKSkRLX0pBVkFfT1BUSU9OUz0iJEpES19KQVZBX09QVElPTlMgLS1hZGQtb3BlbnM9amF2YS5iYXNlL2phdmEudXRpbC5jb25jdXJyZW50PUFMTC1VTk5BTUVEIgpKREtfSkFWQV9PUFRJT05TPSIkSkRLX0pBVkFfT1BUSU9OUyAtLWFkZC1vcGVucz1qYXZhLnJtaS9zdW4ucm1pLnRyYW5zcG9ydD1BTEwtVU5OQU1FRCIKZXhwb3J0IEpES19KQVZBX09QVElPTlMKCiMgLS0tLS0gRXhlY3V0ZSBUaGUgUmVxdWVzdGVkIENvbW1hbmQgLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0KCiMgQnVnemlsbGEgMzc4NDg6IG9ubHkgb3V0cHV0IHRoaXMgaWYgd2UgaGF2ZSBhIFRUWQppZiBbICRoYXZlX3R0eSAtZXEgMSBdOyB0aGVuCiAgZWNobyAiVXNpbmcgQ0FUQUxJTkFfQkFTRTogICAkQ0FUQUxJTkFfQkFTRSIKICBlY2hvICJVc2luZyBDQVRBTElOQV9IT01FOiAgICRDQVRBTElOQV9IT01FIgogIGVjaG8gIlVzaW5nIENBVEFMSU5BX1RNUERJUjogJENBVEFMSU5BX1RNUERJUiIKICBpZiBbICIkMSIgPSAiZGVidWciIF0gOyB0aGVuCiAgICBlY2hvICJVc2luZyBKQVZBX0hPTUU6ICAgICAgICRKQVZBX0hPTUUiCiAgZWxzZQogICAgZWNobyAiVXNpbmcgSlJFX0hPTUU6ICAgICAgICAkSlJFX0hPTUUiCiAgZmkKICBlY2hvICJVc2luZyBDTEFTU1BBVEg6ICAgICAgICRDTEFTU1BBVEgiCiAgZWNobyAiVXNpbmcgQ0FUQUxJTkFfT1BUUzogICAkQ0FUQUxJTkFfT1BUUyIKICBpZiBbICEgLXogIiRDQVRBTElOQV9QSUQiIF07IHRoZW4KICAgIGVjaG8gIlVzaW5nIENBVEFMSU5BX1BJRDogICAgJENBVEFMSU5BX1BJRCIKICBmaQpmaQoKaWYgWyAiJDEiID0gImpwZGEiIF0gOyB0aGVuCiAgaWYgWyAteiAiJEpQREFfVFJBTlNQT1JUIiBdOyB0aGVuCiAgICBKUERBX1RSQU5TUE9SVD0iZHRfc29ja2V0IgogIGZpCiAgaWYgWyAteiAiJEpQREFfQUREUkVTUyIgXTsgdGhlbgogICAgSlBEQV9BRERSRVNTPSJsb2NhbGhvc3Q6ODAwMCIKICBmaQogIGlmIFsgLXogIiRKUERBX1NVU1BFTkQiIF07IHRoZW4KICAgIEpQREFfU1VTUEVORD0ibiIKICBmaQogIGlmIFsgLXogIiRKUERBX09QVFMiIF07IHRoZW4KICAgIEpQREFfT1BUUz0iLWFnZW50bGliOmpkd3A9dHJhbnNwb3J0PSRKUERBX1RSQU5TUE9SVCxhZGRyZXNzPSRKUERBX0FERFJFU1Msc2VydmVyPXksc3VzcGVuZD0kSlBEQV9TVVNQRU5EIgogIGZpCiAgQ0FUQUxJTkFfT1BUUz0iJEpQREFfT1BUUyAkQ0FUQUxJTkFfT1BUUyIKICBzaGlmdApmaQoKaWYgWyAiJDEiID0gImRlYnVnIiBdIDsgdGhlbgogIGlmICRvczQwMDsgdGhlbgogICAgZWNobyAiRGVidWcgY29tbWFuZCBub3QgYXZhaWxhYmxlIG9uIE9TNDAwIgogICAgZXhpdCAxCiAgZWxzZQogICAgc2hpZnQKICAgIGlmIFsgIiQxIiA9ICItc2VjdXJpdHkiIF0gOyB0aGVuCiAgICAgIGlmIFsgJGhhdmVfdHR5IC1lcSAxIF07IHRoZW4KICAgICAgICBlY2hvICJVc2luZyBTZWN1cml0eSBNYW5hZ2VyIgogICAgICBmaQogICAgICBzaGlmdAogICAgICBldmFsIGV4ZWMgIlwiJF9SVU5KREJcIiIgIlwiJENBVEFMSU5BX0xPR0dJTkdfQ09ORklHXCIiICRMT0dHSU5HX01BTkFHRVIgIiRKQVZBX09QVFMiICIkQ0FUQUxJTkFfT1BUUyIgXAogICAgICAgIC1EJEVORE9SU0VEX1BST1A9IiRKQVZBX0VORE9SU0VEX0RJUlMiIFwKICAgICAgICAtY2xhc3NwYXRoICIkQ0xBU1NQQVRIIiBcCiAgICAgICAgLXNvdXJjZXBhdGggIiRDQVRBTElOQV9IT01FIi8uLi8uLi9qYXZhIFwKICAgICAgICAtRGphdmEuc2VjdXJpdHkubWFuYWdlciBcCiAgICAgICAgLURqYXZhLnNlY3VyaXR5LnBvbGljeT09IiRDQVRBTElOQV9CQVNFIi9jb25mL2NhdGFsaW5hLnBvbGljeSBcCiAgICAgICAgLURjYXRhbGluYS5iYXNlPSIkQ0FUQUxJTkFfQkFTRSIgXAogICAgICAgIC1EY2F0YWxpbmEuaG9tZT0iJENBVEFMSU5BX0hPTUUiIFwKICAgICAgICAtRGphdmEuaW8udG1wZGlyPSIkQ0FUQUxJTkFfVE1QRElSIiBcCiAgICAgICAgb3JnLmFwYWNoZS5jYXRhbGluYS5zdGFydHVwLkJvb3RzdHJhcCAiJEAiIHN0YXJ0CiAgICBlbHNlCiAgICAgIGV2YWwgZXhlYyAiXCIkX1JVTkpEQlwiIiAiXCIkQ0FUQUxJTkFfTE9HR0lOR19DT05GSUdcIiIgJExPR0dJTkdfTUFOQUdFUiAiJEpBVkFfT1BUUyIgIiRDQVRBTElOQV9PUFRTIiBcCiAgICAgICAgLUQkRU5ET1JTRURfUFJPUD0iJEpBVkFfRU5ET1JTRURfRElSUyIgXAogICAgICAgIC1jbGFzc3BhdGggIiRDTEFTU1BBVEgiIFwKICAgICAgICAtc291cmNlcGF0aCAiJENBVEFMSU5BX0hPTUUiLy4uLy4uL2phdmEgXAogICAgICAgIC1EY2F0YWxpbmEuYmFzZT0iJENBVEFMSU5BX0JBU0UiIFwKICAgICAgICAtRGNhdGFsaW5hLmhvbWU9IiRDQVRBTElOQV9IT01FIiBcCiAgICAgICAgLURqYXZhLmlvLnRtcGRpcj0iJENBVEFMSU5BX1RNUERJUiIgXAogICAgICAgIG9yZy5hcGFjaGUuY2F0YWxpbmEuc3RhcnR1cC5Cb290c3RyYXAgIiRAIiBzdGFydAogICAgZmkKICBmaQoKZWxpZiBbICIkMSIgPSAicnVuIiBdOyB0aGVuCgogIHNoaWZ0CiAgaWYgWyAiJDEiID0gIi1zZWN1cml0eSIgXSA7IHRoZW4KICAgIGlmIFsgJGhhdmVfdHR5IC1lcSAxIF07IHRoZW4KICAgICAgZWNobyAiVXNpbmcgU2VjdXJpdHkgTWFuYWdlciIKICAgIGZpCiAgICBzaGlmdAogICAgZXZhbCBleGVjICJcIiRfUlVOSkFWQVwiIiAiXCIkQ0FUQUxJTkFfTE9HR0lOR19DT05GSUdcIiIgJExPR0dJTkdfTUFOQUdFUiAiJEpBVkFfT1BUUyIgIiRDQVRBTElOQV9PUFRTIiBcCiAgICAgIC1EJEVORE9SU0VEX1BST1A9IlwiJEpBVkFfRU5ET1JTRURfRElSU1wiIiBcCiAgICAgIC1jbGFzc3BhdGggIlwiJENMQVNTUEFUSFwiIiBcCiAgICAgIC1EamF2YS5zZWN1cml0eS5tYW5hZ2VyIFwKICAgICAgLURqYXZhLnNlY3VyaXR5LnBvbGljeT09IlwiJENBVEFMSU5BX0JBU0UvY29uZi9jYXRhbGluYS5wb2xpY3lcIiIgXAogICAgICAtRGNhdGFsaW5hLmJhc2U9IlwiJENBVEFMSU5BX0JBU0VcIiIgXAogICAgICAtRGNhdGFsaW5hLmhvbWU9IlwiJENBVEFMSU5BX0hPTUVcIiIgXAogICAgICAtRGphdmEuaW8udG1wZGlyPSJcIiRDQVRBTElOQV9UTVBESVJcIiIgXAogICAgICBvcmcuYXBhY2hlLmNhdGFsaW5hLnN0YXJ0dXAuQm9vdHN0cmFwICIkQCIgc3RhcnQKICBlbHNlCiAgICBldmFsIGV4ZWMgIlwiJF9SVU5KQVZBXCIiICJcIiRDQVRBTElOQV9MT0dHSU5HX0NPTkZJR1wiIiAkTE9HR0lOR19NQU5BR0VSICIkSkFWQV9PUFRTIiAiJENBVEFMSU5BX09QVFMiIFwKICAgICAgLUQkRU5ET1JTRURfUFJPUD0iXCIkSkFWQV9FTkRPUlNFRF9ESVJTXCIiIFwKICAgICAgLWNsYXNzcGF0aCAiXCIkQ0xBU1NQQVRIXCIiIFwKICAgICAgLURjYXRhbGluYS5iYXNlPSJcIiRDQVRBTElOQV9CQVNFXCIiIFwKICAgICAgLURjYXRhbGluYS5ob21lPSJcIiRDQVRBTElOQV9IT01FXCIiIFwKICAgICAgLURqYXZhLmlvLnRtcGRpcj0iXCIkQ0FUQUxJTkFfVE1QRElSXCIiIFwKICAgICAgb3JnLmFwYWNoZS5jYXRhbGluYS5zdGFydHVwLkJvb3RzdHJhcCAiJEAiIHN0YXJ0CiAgZmkKCmVsaWYgWyAiJDEiID0gInN0YXJ0IiBdIDsgdGhlbgoKICBpZiBbICEgLXogIiRDQVRBTElOQV9QSUQiIF07IHRoZW4KICAgIGlmIFsgLWYgIiRDQVRBTElOQV9QSUQiIF07IHRoZW4KICAgICAgaWYgWyAtcyAiJENBVEFMSU5BX1BJRCIgXTsgdGhlbgogICAgICAgIGVjaG8gIkV4aXN0aW5nIFBJRCBmaWxlIGZvdW5kIGR1cmluZyBzdGFydC4iCiAgICAgICAgaWYgWyAtciAiJENBVEFMSU5BX1BJRCIgXTsgdGhlbgogICAgICAgICAgUElEPWBjYXQgIiRDQVRBTElOQV9QSUQiYAogICAgICAgICAgcHMgLXAgJFBJRCA+L2Rldi9udWxsIDI+JjEKICAgICAgICAgIGlmIFsgJD8gLWVxIDAgXSA7IHRoZW4KICAgICAgICAgICAgZWNobyAiVG9tY2F0IGFwcGVhcnMgdG8gc3RpbGwgYmUgcnVubmluZyB3aXRoIFBJRCAkUElELiBTdGFydCBhYm9ydGVkLiIKICAgICAgICAgICAgZWNobyAiSWYgdGhlIGZvbGxvd2luZyBwcm9jZXNzIGlzIG5vdCBhIFRvbWNhdCBwcm9jZXNzLCByZW1vdmUgdGhlIFBJRCBmaWxlIGFuZCB0cnkgYWdhaW46IgogICAgICAgICAgICBwcyAtZiAtcCAkUElECiAgICAgICAgICAgIGV4aXQgMQogICAgICAgICAgZWxzZQogICAgICAgICAgICBlY2hvICJSZW1vdmluZy9jbGVhcmluZyBzdGFsZSBQSUQgZmlsZS4iCiAgICAgICAgICAgIHJtIC1mICIkQ0FUQUxJTkFfUElEIiA+L2Rldi9udWxsIDI+JjEKICAgICAgICAgICAgaWYgWyAkPyAhPSAwIF07IHRoZW4KICAgICAgICAgICAgICBpZiBbIC13ICIkQ0FUQUxJTkFfUElEIiBdOyB0aGVuCiAgICAgICAgICAgICAgICBjYXQgL2Rldi9udWxsID4gIiRDQVRBTElOQV9QSUQiCiAgICAgICAgICAgICAgZWxzZQogICAgICAgICAgICAgICAgZWNobyAiVW5hYmxlIHRvIHJlbW92ZSBvciBjbGVhciBzdGFsZSBQSUQgZmlsZS4gU3RhcnQgYWJvcnRlZC4iCiAgICAgICAgICAgICAgICBleGl0IDEKICAgICAgICAgICAgICBmaQogICAgICAgICAgICBmaQogICAgICAgICAgZmkKICAgICAgICBlbHNlCiAgICAgICAgICBlY2hvICJVbmFibGUgdG8gcmVhZCBQSUQgZmlsZS4gU3RhcnQgYWJvcnRlZC4iCiAgICAgICAgICBleGl0IDEKICAgICAgICBmaQogICAgICBlbHNlCiAgICAgICAgcm0gLWYgIiRDQVRBTElOQV9QSUQiID4vZGV2L251bGwgMj4mMQogICAgICAgIGlmIFsgJD8gIT0gMCBdOyB0aGVuCiAgICAgICAgICBpZiBbICEgLXcgIiRDQVRBTElOQV9QSUQiIF07IHRoZW4KICAgICAgICAgICAgZWNobyAiVW5hYmxlIHRvIHJlbW92ZSBvciB3cml0ZSB0byBlbXB0eSBQSUQgZmlsZS4gU3RhcnQgYWJvcnRlZC4iCiAgICAgICAgICAgIGV4aXQgMQogICAgICAgICAgZmkKICAgICAgICBmaQogICAgICBmaQogICAgZmkKICBmaQoKICBzaGlmdAogIGlmIFsgLXogIiRDQVRBTElOQV9PVVRfQ01EIiBdIDsgdGhlbgogICAgdG91Y2ggIiRDQVRBTElOQV9PVVQiCiAgZWxzZQogICAgaWYgWyAhIC1lICIkQ0FUQUxJTkFfT1VUIiBdOyB0aGVuCiAgICAgIGlmICEgbWtmaWZvICIkQ0FUQUxJTkFfT1VUIjsgdGhlbgogICAgICAgIGVjaG8gImNhbm5vdCBjcmVhdGUgbmFtZWQgcGlwZSAkQ0FUQUxJTkFfT1VULiBTdGFydCBhYm9ydGVkLiIKICAgICAgICBleGl0IDEKICAgICAgZmkKICAgIGVsaWYgWyAhIC1wICIkQ0FUQUxJTkFfT1VUIiBdOyB0aGVuCiAgICAgIGVjaG8gIiRDQVRBTElOQV9PVVQgZXhpc3RzIGFuZCBpcyBub3QgYSBuYW1lZCBwaXBlLiBTdGFydCBhYm9ydGVkLiIKICAgICAgZXhpdCAxCiAgICBmaQogICAgJENBVEFMSU5BX09VVF9DTUQgPCIkQ0FUQUxJTkFfT1VUIiAmCiAgZmkKICBpZiBbICIkMSIgPSAiLXNlY3VyaXR5IiBdIDsgdGhlbgogICAgaWYgWyAkaGF2ZV90dHkgLWVxIDEgXTsgdGhlbgogICAgICBlY2hvICJVc2luZyBTZWN1cml0eSBNYW5hZ2VyIgogICAgZmkKICAgIHNoaWZ0CiAgICBldmFsICRfTk9IVVAgIlwiJF9SVU5KQVZBXCIiICJcIiRDQVRBTElOQV9MT0dHSU5HX0NPTkZJR1wiIiAkTE9HR0lOR19NQU5BR0VSICIkSkFWQV9PUFRTIiAiJENBVEFMSU5BX09QVFMiIFwKICAgICAgLUQkRU5ET1JTRURfUFJPUD0iXCIkSkFWQV9FTkRPUlNFRF9ESVJTXCIiIFwKICAgICAgLWNsYXNzcGF0aCAiXCIkQ0xBU1NQQVRIXCIiIFwKICAgICAgLURqYXZhLnNlY3VyaXR5Lm1hbmFnZXIgXAogICAgICAtRGphdmEuc2VjdXJpdHkucG9saWN5PT0iXCIkQ0FUQUxJTkFfQkFTRS9jb25mL2NhdGFsaW5hLnBvbGljeVwiIiBcCiAgICAgIC1EY2F0YWxpbmEuYmFzZT0iXCIkQ0FUQUxJTkFfQkFTRVwiIiBcCiAgICAgIC1EY2F0YWxpbmEuaG9tZT0iXCIkQ0FUQUxJTkFfSE9NRVwiIiBcCiAgICAgIC1EamF2YS5pby50bXBkaXI9IlwiJENBVEFMSU5BX1RNUERJUlwiIiBcCiAgICAgIG9yZy5hcGFjaGUuY2F0YWxpbmEuc3RhcnR1cC5Cb290c3RyYXAgIiRAIiBzdGFydCBcCiAgICAgID4+ICIkQ0FUQUxJTkFfT1VUIiAyPiYxICImIgoKICBlbHNlCiAgICBldmFsICRfTk9IVVAgIlwiJF9SVU5KQVZBXCIiICJcIiRDQVRBTElOQV9MT0dHSU5HX0NPTkZJR1wiIiAkTE9HR0lOR19NQU5BR0VSICIkSkFWQV9PUFRTIiAiJENBVEFMSU5BX09QVFMiIFwKICAgICAgLUQkRU5ET1JTRURfUFJPUD0iXCIkSkFWQV9FTkRPUlNFRF9ESVJTXCIiIFwKICAgICAgLWNsYXNzcGF0aCAiXCIkQ0xBU1NQQVRIXCIiIFwKICAgICAgLURjYXRhbGluYS5iYXNlPSJcIiRDQVRBTElOQV9CQVNFXCIiIFwKICAgICAgLURjYXRhbGluYS5ob21lPSJcIiRDQVRBTElOQV9IT01FXCIiIFwKICAgICAgLURqYXZhLmlvLnRtcGRpcj0iXCIkQ0FUQUxJTkFfVE1QRElSXCIiIFwKICAgICAgb3JnLmFwYWNoZS5jYXRhbGluYS5zdGFydHVwLkJvb3RzdHJhcCAiJEAiIHN0YXJ0IFwKICAgICAgPj4gIiRDQVRBTElOQV9PVVQiIDI+JjEgIiYiCgogIGZpCgogIGlmIFsgISAteiAiJENBVEFMSU5BX1BJRCIgXTsgdGhlbgogICAgZWNobyAkISA+ICIkQ0FUQUxJTkFfUElEIgogIGZpCgogIGVjaG8gIlRvbWNhdCBzdGFydGVkLiIKCmVsaWYgWyAiJDEiID0gInN0b3AiIF0gOyB0aGVuCgogIHNoaWZ0CgogIFNMRUVQPTUKICBpZiBbICEgLXogIiQxIiBdOyB0aGVuCiAgICBlY2hvICQxIHwgZ3JlcCAiW14wLTldIiA+L2Rldi9udWxsIDI+JjEKICAgIGlmIFsgJD8gLWd0IDAgXTsgdGhlbgogICAgICBTTEVFUD0kMQogICAgICBzaGlmdAogICAgZmkKICBmaQoKICBGT1JDRT0wCiAgaWYgWyAiJDEiID0gIi1mb3JjZSIgXTsgdGhlbgogICAgc2hpZnQKICAgIEZPUkNFPTEKICBmaQoKICBpZiBbICEgLXogIiRDQVRBTElOQV9QSUQiIF07IHRoZW4KICAgIGlmIFsgLWYgIiRDQVRBTElOQV9QSUQiIF07IHRoZW4KICAgICAgaWYgWyAtcyAiJENBVEFMSU5BX1BJRCIgXTsgdGhlbgogICAgICAgIGtpbGwgLTAgYGNhdCAiJENBVEFMSU5BX1BJRCJgID4vZGV2L251bGwgMj4mMQogICAgICAgIGlmIFsgJD8gLWd0IDAgXTsgdGhlbgogICAgICAgICAgZWNobyAiUElEIGZpbGUgZm91bmQgYnV0IGVpdGhlciBubyBtYXRjaGluZyBwcm9jZXNzIHdhcyBmb3VuZCBvciB0aGUgY3VycmVudCB1c2VyIGRvZXMgbm90IGhhdmUgcGVybWlzc2lvbiB0byBzdG9wIHRoZSBwcm9jZXNzLiBTdG9wIGFib3J0ZWQuIgogICAgICAgICAgZXhpdCAxCiAgICAgICAgZmkKICAgICAgZWxzZQogICAgICAgIGVjaG8gIlBJRCBmaWxlIGlzIGVtcHR5IGFuZCBoYXMgYmVlbiBpZ25vcmVkLiIKICAgICAgZmkKICAgIGVsc2UKICAgICAgZWNobyAiXCRDQVRBTElOQV9QSUQgd2FzIHNldCBidXQgdGhlIHNwZWNpZmllZCBmaWxlIGRvZXMgbm90IGV4aXN0LiBJcyBUb21jYXQgcnVubmluZz8gU3RvcCBhYm9ydGVkLiIKICAgICAgZXhpdCAxCiAgICBmaQogIGZpCgogIGV2YWwgIlwiJF9SVU5KQVZBXCIiICRMT0dHSU5HX01BTkFHRVIgIiRKQVZBX09QVFMiIFwKICAgIC1EJEVORE9SU0VEX1BST1A9IlwiJEpBVkFfRU5ET1JTRURfRElSU1wiIiBcCiAgICAtY2xhc3NwYXRoICJcIiRDTEFTU1BBVEhcIiIgXAogICAgLURjYXRhbGluYS5iYXNlPSJcIiRDQVRBTElOQV9CQVNFXCIiIFwKICAgIC1EY2F0YWxpbmEuaG9tZT0iXCIkQ0FUQUxJTkFfSE9NRVwiIiBcCiAgICAtRGphdmEuaW8udG1wZGlyPSJcIiRDQVRBTElOQV9UTVBESVJcIiIgXAogICAgb3JnLmFwYWNoZS5jYXRhbGluYS5zdGFydHVwLkJvb3RzdHJhcCAiJEAiIHN0b3AKCiAgIyBzdG9wIGZhaWxlZC4gU2h1dGRvd24gcG9ydCBkaXNhYmxlZD8gVHJ5IGEgbm9ybWFsIGtpbGwuCiAgaWYgWyAkPyAhPSAwIF07IHRoZW4KICAgIGlmIFsgISAteiAiJENBVEFMSU5BX1BJRCIgXTsgdGhlbgogICAgICBlY2hvICJUaGUgc3RvcCBjb21tYW5kIGZhaWxlZC4gQXR0ZW1wdGluZyB0byBzaWduYWwgdGhlIHByb2Nlc3MgdG8gc3RvcCB0aHJvdWdoIE9TIHNpZ25hbC4iCiAgICAgIGtpbGwgLTE1IGBjYXQgIiRDQVRBTElOQV9QSUQiYCA+L2Rldi9udWxsIDI+JjEKICAgIGZpCiAgZmkKCiAgaWYgWyAhIC16ICIkQ0FUQUxJTkFfUElEIiBdOyB0aGVuCiAgICBpZiBbIC1mICIkQ0FUQUxJTkFfUElEIiBdOyB0aGVuCiAgICAgIHdoaWxlIFsgJFNMRUVQIC1nZSAwIF07IGRvCiAgICAgICAga2lsbCAtMCBgY2F0ICIkQ0FUQUxJTkFfUElEImAgPi9kZXYvbnVsbCAyPiYxCiAgICAgICAgaWYgWyAkPyAtZ3QgMCBdOyB0aGVuCiAgICAgICAgICBybSAtZiAiJENBVEFMSU5BX1BJRCIgPi9kZXYvbnVsbCAyPiYxCiAgICAgICAgICBpZiBbICQ/ICE9IDAgXTsgdGhlbgogICAgICAgICAgICBpZiBbIC13ICIkQ0FUQUxJTkFfUElEIiBdOyB0aGVuCiAgICAgICAgICAgICAgY2F0IC9kZXYvbnVsbCA+ICIkQ0FUQUxJTkFfUElEIgogICAgICAgICAgICAgICMgSWYgVG9tY2F0IGhhcyBzdG9wcGVkIGRvbid0IHRyeSBhbmQgZm9yY2UgYSBzdG9wIHdpdGggYW4gZW1wdHkgUElEIGZpbGUKICAgICAgICAgICAgICBGT1JDRT0wCiAgICAgICAgICAgIGVsc2UKICAgICAgICAgICAgICBlY2hvICJUaGUgUElEIGZpbGUgY291bGQgbm90IGJlIHJlbW92ZWQgb3IgY2xlYXJlZC4iCiAgICAgICAgICAgIGZpCiAgICAgICAgICBmaQogICAgICAgICAgZWNobyAiVG9tY2F0IHN0b3BwZWQuIgogICAgICAgICAgYnJlYWsKICAgICAgICBmaQogICAgICAgIGlmIFsgJFNMRUVQIC1ndCAwIF07IHRoZW4KICAgICAgICAgIHNsZWVwIDEKICAgICAgICBmaQogICAgICAgIGlmIFsgJFNMRUVQIC1lcSAwIF07IHRoZW4KICAgICAgICAgIGVjaG8gIlRvbWNhdCBkaWQgbm90IHN0b3AgaW4gdGltZS4iCiAgICAgICAgICBpZiBbICRGT1JDRSAtZXEgMCBdOyB0aGVuCiAgICAgICAgICAgIGVjaG8gIlBJRCBmaWxlIHdhcyBub3QgcmVtb3ZlZC4iCiAgICAgICAgICBmaQogICAgICAgICAgZWNobyAiVG8gYWlkIGRpYWdub3N0aWNzIGEgdGhyZWFkIGR1bXAgaGFzIGJlZW4gd3JpdHRlbiB0byBzdGFuZGFyZCBvdXQuIgogICAgICAgICAga2lsbCAtMyBgY2F0ICIkQ0FUQUxJTkFfUElEImAKICAgICAgICBmaQogICAgICAgIFNMRUVQPWBleHByICRTTEVFUCAtIDEgYAogICAgICBkb25lCiAgICBmaQogIGZpCgogIEtJTExfU0xFRVBfSU5URVJWQUw9NQogIGlmIFsgJEZPUkNFIC1lcSAxIF07IHRoZW4KICAgIGlmIFsgLXogIiRDQVRBTElOQV9QSUQiIF07IHRoZW4KICAgICAgZWNobyAiS2lsbCBmYWlsZWQ6IFwkQ0FUQUxJTkFfUElEIG5vdCBzZXQiCiAgICBlbHNlCiAgICAgIGlmIFsgLWYgIiRDQVRBTElOQV9QSUQiIF07IHRoZW4KICAgICAgICBQSUQ9YGNhdCAiJENBVEFMSU5BX1BJRCJgCiAgICAgICAgZWNobyAiS2lsbGluZyBUb21jYXQgd2l0aCB0aGUgUElEOiAkUElEIgogICAgICAgIGtpbGwgLTkgJFBJRAogICAgICAgIHdoaWxlIFsgJEtJTExfU0xFRVBfSU5URVJWQUwgLWdlIDAgXTsgZG8KICAgICAgICAgICAga2lsbCAtMCBgY2F0ICIkQ0FUQUxJTkFfUElEImAgPi9kZXYvbnVsbCAyPiYxCiAgICAgICAgICAgIGlmIFsgJD8gLWd0IDAgXTsgdGhlbgogICAgICAgICAgICAgICAgcm0gLWYgIiRDQVRBTElOQV9QSUQiID4vZGV2L251bGwgMj4mMQogICAgICAgICAgICAgICAgaWYgWyAkPyAhPSAwIF07IHRoZW4KICAgICAgICAgICAgICAgICAgICBpZiBbIC13ICIkQ0FUQUxJTkFfUElEIiBdOyB0aGVuCiAgICAgICAgICAgICAgICAgICAgICAgIGNhdCAvZGV2L251bGwgPiAiJENBVEFMSU5BX1BJRCIKICAgICAgICAgICAgICAgICAgICBlbHNlCiAgICAgICAgICAgICAgICAgICAgICAgIGVjaG8gIlRoZSBQSUQgZmlsZSBjb3VsZCBub3QgYmUgcmVtb3ZlZC4iCiAgICAgICAgICAgICAgICAgICAgZmkKICAgICAgICAgICAgICAgIGZpCiAgICAgICAgICAgICAgICBlY2hvICJUaGUgVG9tY2F0IHByb2Nlc3MgaGFzIGJlZW4ga2lsbGVkLiIKICAgICAgICAgICAgICAgIGJyZWFrCiAgICAgICAgICAgIGZpCiAgICAgICAgICAgIGlmIFsgJEtJTExfU0xFRVBfSU5URVJWQUwgLWd0IDAgXTsgdGhlbgogICAgICAgICAgICAgICAgc2xlZXAgMQogICAgICAgICAgICBmaQogICAgICAgICAgICBLSUxMX1NMRUVQX0lOVEVSVkFMPWBleHByICRLSUxMX1NMRUVQX0lOVEVSVkFMIC0gMSBgCiAgICAgICAgZG9uZQogICAgICAgIGlmIFsgJEtJTExfU0xFRVBfSU5URVJWQUwgLWx0IDAgXTsgdGhlbgogICAgICAgICAgICBlY2hvICJUb21jYXQgaGFzIG5vdCBiZWVuIGtpbGxlZCBjb21wbGV0ZWx5IHlldC4gVGhlIHByb2Nlc3MgbWlnaHQgYmUgd2FpdGluZyBvbiBzb21lIHN5c3RlbSBjYWxsIG9yIG1pZ2h0IGJlIFVOSU5URVJSVVBUSUJMRS4iCiAgICAgICAgZmkKICAgICAgZmkKICAgIGZpCiAgZmkKCmVsaWYgWyAiJDEiID0gImNvbmZpZ3Rlc3QiIF0gOyB0aGVuCgogICAgZXZhbCAiXCIkX1JVTkpBVkFcIiIgJExPR0dJTkdfTUFOQUdFUiAiJEpBVkFfT1BUUyIgXAogICAgICAtRCRFTkRPUlNFRF9QUk9QPSJcIiRKQVZBX0VORE9SU0VEX0RJUlNcIiIgXAogICAgICAtY2xhc3NwYXRoICJcIiRDTEFTU1BBVEhcIiIgXAogICAgICAtRGNhdGFsaW5hLmJhc2U9IlwiJENBVEFMSU5BX0JBU0VcIiIgXAogICAgICAtRGNhdGFsaW5hLmhvbWU9IlwiJENBVEFMSU5BX0hPTUVcIiIgXAogICAgICAtRGphdmEuaW8udG1wZGlyPSJcIiRDQVRBTElOQV9UTVBESVJcIiIgXAogICAgICBvcmcuYXBhY2hlLmNhdGFsaW5hLnN0YXJ0dXAuQm9vdHN0cmFwIGNvbmZpZ3Rlc3QKICAgIHJlc3VsdD0kPwogICAgaWYgWyAkcmVzdWx0IC1uZSAwIF07IHRoZW4KICAgICAgICBlY2hvICJDb25maWd1cmF0aW9uIGVycm9yIGRldGVjdGVkISIKICAgIGZpCiAgICBleGl0ICRyZXN1bHQKCmVsaWYgWyAiJDEiID0gInZlcnNpb24iIF0gOyB0aGVuCgogICBldmFsICJcIiRfUlVOSkFWQVwiIiAiJEpBVkFfT1BUUyIgXAogICAgICAgICAtY2xhc3NwYXRoICJcIiRDQVRBTElOQV9IT01FL2xpYi9jYXRhbGluYS5qYXJcIiIgXAogICAgICAgICBvcmcuYXBhY2hlLmNhdGFsaW5hLnV0aWwuU2VydmVySW5mbwoKZWxzZQoKICBlY2hvICJVc2FnZTogY2F0YWxpbmEuc2ggKCBjb21tYW5kcyAuLi4gKSIKICBlY2hvICJjb21tYW5kczoiCiAgaWYgJG9zNDAwOyB0aGVuCiAgICBlY2hvICIgIGRlYnVnICAgICAgICAgICAgIFN0YXJ0IENhdGFsaW5hIGluIGEgZGVidWdnZXIgKG5vdCBhdmFpbGFibGUgb24gT1M0MDApIgogICAgZWNobyAiICBkZWJ1ZyAtc2VjdXJpdHkgICBEZWJ1ZyBDYXRhbGluYSB3aXRoIGEgc2VjdXJpdHkgbWFuYWdlciAobm90IGF2YWlsYWJsZSBvbiBPUzQwMCkiCiAgZWxzZQogICAgZWNobyAiICBkZWJ1ZyAgICAgICAgICAgICBTdGFydCBDYXRhbGluYSBpbiBhIGRlYnVnZ2VyIgogICAgZWNobyAiICBkZWJ1ZyAtc2VjdXJpdHkgICBEZWJ1ZyBDYXRhbGluYSB3aXRoIGEgc2VjdXJpdHkgbWFuYWdlciIKICBmaQogIGVjaG8gIiAganBkYSBzdGFydCAgICAgICAgU3RhcnQgQ2F0YWxpbmEgdW5kZXIgSlBEQSBkZWJ1Z2dlciIKICBlY2hvICIgIHJ1biAgICAgICAgICAgICAgIFN0YXJ0IENhdGFsaW5hIGluIHRoZSBjdXJyZW50IHdpbmRvdyIKICBlY2hvICIgIHJ1biAtc2VjdXJpdHkgICAgIFN0YXJ0IGluIHRoZSBjdXJyZW50IHdpbmRvdyB3aXRoIHNlY3VyaXR5IG1hbmFnZXIiCiAgZWNobyAiICBzdGFydCAgICAgICAgICAgICBTdGFydCBDYXRhbGluYSBpbiBhIHNlcGFyYXRlIHdpbmRvdyIKICBlY2hvICIgIHN0YXJ0IC1zZWN1cml0eSAgIFN0YXJ0IGluIGEgc2VwYXJhdGUgd2luZG93IHdpdGggc2VjdXJpdHkgbWFuYWdlciIKICBlY2hvICIgIHN0b3AgICAgICAgICAgICAgIFN0b3AgQ2F0YWxpbmEsIHdhaXRpbmcgdXAgdG8gNSBzZWNvbmRzIGZvciB0aGUgcHJvY2VzcyB0byBlbmQiCiAgZWNobyAiICBzdG9wIG4gICAgICAgICAgICBTdG9wIENhdGFsaW5hLCB3YWl0aW5nIHVwIHRvIG4gc2Vjb25kcyBmb3IgdGhlIHByb2Nlc3MgdG8gZW5kIgogIGVjaG8gIiAgc3RvcCAtZm9yY2UgICAgICAgU3RvcCBDYXRhbGluYSwgd2FpdCB1cCB0byA1IHNlY29uZHMgYW5kIHRoZW4gdXNlIGtpbGwgLUtJTEwgaWYgc3RpbGwgcnVubmluZyIKICBlY2hvICIgIHN0b3AgbiAtZm9yY2UgICAgIFN0b3AgQ2F0YWxpbmEsIHdhaXQgdXAgdG8gbiBzZWNvbmRzIGFuZCB0aGVuIHVzZSBraWxsIC1LSUxMIGlmIHN0aWxsIHJ1bm5pbmciCiAgZWNobyAiICBjb25maWd0ZXN0ICAgICAgICBSdW4gYSBiYXNpYyBzeW50YXggY2hlY2sgb24gc2VydmVyLnhtbCAtIGNoZWNrIGV4aXQgY29kZSBmb3IgcmVzdWx0IgogIGVjaG8gIiAgdmVyc2lvbiAgICAgICAgICAgV2hhdCB2ZXJzaW9uIG9mIHRvbWNhdCBhcmUgeW91IHJ1bm5pbmc/IgogIGVjaG8gIk5vdGU6IFdhaXRpbmcgZm9yIHRoZSBwcm9jZXNzIHRvIGVuZCBhbmQgdXNlIG9mIHRoZSAtZm9yY2Ugb3B0aW9uIHJlcXVpcmUgdGhhdCBcJENBVEFMSU5BX1BJRCBpcyBkZWZpbmVkIgogIGV4aXQgMQoKZmkK
ccv2.file.override.71.path=/opt/tomcat/bin/catalina-original.sh

• After the change of the service configuration, the service should be restarted to be effective
• To confirm the environment variable has been set, you can execute the following Groovy script on HAC on the right service.

def envVars = System.getenv()
envVars.each { key, value ->
    if(key.equals("MALLOC_ARENA_MAX"))
    {
        println("${key} = ${value}")
    }
}

[ductran]

Hi @YannickRobin,

Thanks for sharing, I went thru the article and have 2 questions. It would be great that you could follow up please:

- Given example of "export MALLOC_ARENA_MAX=8" does it mean this node is having 4 cores? I am confused that in before those settings your environment has been configured with 10 cores, which supposes to be resulting to "export MALLOC_ARENA_MAX=20"? Can I confirm that "10 cores" and "export MALLOC_ARENA_MAX=8" are different environments?

- I understand that we are about to intervene MALLOC_ARENA_MAX by setting up 2 catalina versions. Decoding base 64 value from ccv2.file.override.71.content I could see the the content is exactly identical with our catalina.sh as script below (scripted on hac)

import org.apache.commons.io.FileUtils;
String contents = FileUtils.readFileToString(new File("/opt/tomcat/bin/catalina.sh"), "UTF-8")

Now my concern is, if we force ccv2 to maintain override version of /opt/tomcat/bin/catalina-original.sh does that mean we are giving up the benefit of new catalina.sh would be aligned automatically once SAP release new update? Do we have any other way that allows to set that variable and not to touch on original version?

Thanks,

Duc.

[YannickRobin]

Hello Duc,

I've updated the post. Hope it clarifies your comments.

Yes, it requires changing the content in case of Tomcat upgrade.

Thanks,

Yannick

[ductran]

Thanks @YannickRobin for quick response. Now all questions are well explained.

Duc.

[laurent-malvert]

Very interesting @YannickRobin ! 👏 Thanks for the write-up 🙂

Quick question (and forgive me if that's stupid as I've been out of the Commerce loop and the CCv2 stack specifics for some time): As the issue seems to be caused by the default arena sizing by glibc's allocator, as you point out in the article, have others allocators been considered ?

I know other cloud providers favor other allocators over glibc's. A few eons ago, tcmalloc was often mentioned as an alternative, or more recently mimalloc.

Maybe not something anyone can meddle with on CCv2 deployments, but maybe a point to carefully audit with the infrastructure team ?

[YannickRobin]

Indeed the problem has been escalated to Product team and they're evaluating various solutions to fix the problem. Using malloc alternative is part of the discussion.

This being said changing malloc arena size seems easy to rollout globally with limited drawback.

[ductran]

Hi Yannick,

Glad to know that Product team is following up, thanks for the update.

In the mean time we will try to tune down the malloc arena first. Please update us again if SAP comes up with permanent fix.

Thanks,

Duc.

[ductran]

Hi,

I has applied the bash script into our cloud commerce, it seems to be working well. I just want to share how I can facilitate the configurations with less mistake.

From HAC, run the script via Platform -> Scripting languages.

Before you run please be mindful 2 things:

- This script should be run after you have upgraded Tomcat. As Yannick shared we have to update manually when Tomcat version is changed.

- If you want to make the settings for different application nodes (eg: running this script on backoffice but to produce the settings for backgroundprocessing), the cores might be different. In this case you can check the number of cores in Dynatrace, under Infrastructure dashboard.

// recommendation from SAP
def MALLOC_ARENA_FACTORS = 2;
//If you would like to get number of cores automatically from impacted application nodes
def cores = Runtime.getRuntime().availableProcessors();
//If you want to generate for other nodes from any Hybris HAC
//def cores = 10;
def maxMallocArenas = cores * MALLOC_ARENA_FACTORS;

def catalinaOverridePlainText = new StringBuilder();
catalinaOverridePlainText.append("#!/usr/bin/env bash\n");
catalinaOverridePlainText.append("set -e\n");
catalinaOverridePlainText.append(String.format("echo \"Set MALLOC ARENA MAX to %d\"\n", maxMallocArenas));
catalinaOverridePlainText.append(String.format("export MALLOC_ARENA_MAX=%d\n", maxMallocArenas));
catalinaOverridePlainText.append("chown hybris /opt/tomcat/bin/catalina-original.sh\n");
catalinaOverridePlainText.append("chmod +x /opt/tomcat/bin/catalina-original.sh\n");
catalinaOverridePlainText.append("exec /opt/tomcat/bin/catalina-original.sh run\n");

//if you are going to run on local or on-prems it is under /bin/platform/resources/tomcat-xxx/bin
def originalCatalinaPlainText = org.apache.commons.io.FileUtils.readFileToString(new File("/opt/tomcat/bin/catalina.sh"), "UTF-8");

println "Your override MALLOC_ARENA_MAX script is going to be:";
println catalinaOverridePlainText.toString();
println "You might want to copy those lines onto ccv2 portal - under environment settings";
println "##############";
println String.format("ccv2.file.override.70.content=%s", Base64.getEncoder().encodeToString(catalinaOverridePlainText.toString().getBytes("UTF-8")))
println "ccv2.file.override.70.path=/opt/tomcat/bin/catalina.sh";
println String.format("ccv2.file.override.71.content=%s", Base64.getEncoder().encodeToString(originalCatalinaPlainText.toString().getBytes("UTF-8")))
println "ccv2.file.override.71.path=/opt/tomcat/bin/catalina-original.sh";
println "##############";

 Hac might creates you something like this

Now you need to copy content amongst hashes and apply them onto ccv2 portal. You might want to check the plain content again from https://www.base64decode.org.

Hope it helps,

Duc.