Puzzling OOM situation

[Sapienstar]

Hello,

I have a test S4 system and I am having some puzzling OOM situations.

OK, this is a test system with limited memory (64 GB), however the oom trace file doesn't help.

This is compile server trace is below.There are for other servers too.

Kindly asking your support with any ideas. Thanks

[OS_MEMORY]  Operating system information about memory: (2022-04-12 14:07:07 468 Local)
Maximum size of the process's virtual memory: soft limit=18446744073709551615, hard limit=18446744073709551615
/proc/meminfo:
MemTotal:       66731648 kB
MemFree:         1432128 kB
MemAvailable:    6723648 kB
Buffers:           32384 kB
Cached:         11084032 kB
SwapCached:        16256 kB
Active:         59085952 kB
Inactive:        4960832 kB
Active(anon):   56102976 kB
Inactive(anon):   447296 kB
Active(file):    2982976 kB
Inactive(file):  4513536 kB
Unevictable:       31232 kB
Mlocked:           31232 kB
SwapTotal:       2097088 kB
SwapFree:        1754304 kB
Dirty:              5440 kB
Writeback:             0 kB
AnonPages:      52955200 kB
Mapped:          4675200 kB
Shmem:           3607360 kB
KReclaimable:     376896 kB
Slab:             930304 kB
SReclaimable:     376896 kB
SUnreclaim:       553408 kB
KernelStack:       28192 kB
PageTables:        68800 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    35462912 kB
Committed_AS:   57347328 kB
VmallocTotal:   549755813888 kB
VmallocUsed:      120128 kB
VmallocChunk:          0 kB
Percpu:            20480 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:      16384 kB
Hugetlb:               0 kB

[MEMORY_OOM]  Information about current out of memory situation: (2022-04-12 14:07:07 477 Local)
OUT OF MEMORY occurred.
Host: testhana01
Executable: hdbcompileserver
PID: 25146
Failed to allocate 64.04kb (65584b).
Allocation failure type: GLOBAL_ALLOCATION_LIMIT (early exit)

GLOBAL_MAX_ALLOCATION_LIMIT=57.27gb (61499834368b)
SHARED_MEMORY=3.40gb (3658350592b)
OOM_RESERVATION_ALLOCATORS=96.18mb (100859904b)
CODE_SIZE=3.26gb (3510632448b)
IPMM_RESERVATION=0b
Sum TPAL (including shm, code size and IPMM reservation but not IPMM OOM reservation)=57.27gb (61499834368b) (cached sum=57.27gb (61499834368b)), emergency reservation=128mb (134217728b)
#processes=9, weight sum=900, cached weight sum=900, #checks for cached allocation limit sum=1682881, #terminated processes since last reset=10
Early exits are currently allowed. 
66531 early exits since creation of IPMM, reclaim-memory-counter=174209. Reclaim memory 174207 failed, only 56688640b were free, 54327812343b were used.
Last reclaim memory started to wait for IPMM lock since 2022-04-12 14:07:06.097.
Last reclaim memory got IPMM lock at 2022-04-12 14:07:07.362.
User ids respected in shm calculation: [1003]
Parallel memory reclaim tasks are activated.
This is an opt build.

[HakanHaslaman]

Can you share us the information please, which exact HANA DB Version with SP and PL you are using?
Could you please also share the top allocators of the exclusive_size_in_use?

[Sapienstar]

Hello hakan.haslaman ,

thank you for your support.

M_HEAP_MEMORY[0] - (id= 60, type= public:remote/multi/remote_datacenter, flags= AddVolumeID AddStatisticsID Crashdump RTEdump, priv= Default)
"STATISTICS_ID";"CATEGORY";"DEPTH";"INCLUSIVE_SIZE_IN_USE [byte]";"INCLUSIVE_COUNT_IN_USE";"INCLUSIVE_ALLOCATED_SIZE [byte]";"INCLUSIVE_DEALLOCATED_SIZE [byte]";"INCLUSIVE_ALLOCATED_COUNT";"INCLUSIVE_DEALLOCATED_COUNT";"INCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE [byte]";"INCLUSIVE_PEAK_ALLOCATION_SIZE [byte]";"INCLUSIVE_LIMIT [byte]";"INCLUSIVE_IN_USE_INTEGRAL";"EXCLUSIVE_SIZE_IN_USE [byte]";"EXCLUSIVE_COUNT_IN_USE";"EXCLUSIVE_ALLOCATED_SIZE [byte]";"EXCLUSIVE_DEALLOCATED_SIZE [byte]";"EXCLUSIVE_ALLOCATED_COUNT";"EXCLUSIVE_DEALLOCATED_COUNT";"EXCLUSIVE_MAX_SINGLE_ALLOCATION_SIZE [byte]";"EXCLUSIVE_PEAK_ALLOCATION_SIZE [byte]";"EXCLUSIVE_ALLOC_ERRORS";"EXCLUSIVE_IN_USE_INTEGRAL";"MALLOC_PROXY_CACHE_MISSES";"FLAGS";"COMPONENT"
5;"/";0;43854485;49457;110117540;66263055;96775;47318;2359296;43913061;18446744073709551615;0;0;0;0;0;0;0;0;0;0;0;0;"(none)";"System"
8;"AllocateOnlyAllocator-unlimited";1;2380816;1328;4414144;2033328;2628;1300;238592;2380816;18446744073709551615;0;309056;158;2262808;1953752;158;0;238592;313104;0;0;0;"preventcheck";"System"
1015;"AllocateOnlyAllocator-unlimited/FLA-UL<24592,1>";2;1623072;66;1623072;0;66;0;24592;1623072;18446744073709551615;0;0;0;0;0;0;0;0;0;0;0;0;"(none)";"System"
1016;"AllocateOnlyAllocator-unlimited/FLA-UL<24592,1>/MemoryMapLevel3Nodes";3;1623072;66;1623072;0;66;0;24592;1623072;18446744073709551615;0;1623072;66;1623072;0;66;0;24592;1623072;0;0;0;"(none)";"System"
1017;"Pool";1;12839752;47900;75755743;62915991;93888;45988;1572880;12935192;18446744073709551615;0;21328;84;21328;0;84;0;256;21328;0;0;0;"(none)";"System"
2884;"Pool/TraceAllocator";2;2104528;51;2109888;5360;99;48;1048576;2104528;18446744073709551615;0;2104528;51;2109888;5360;99;48;1048576;2104528;0;0;0;"(none)";"Monitoring & Statistical Data"
1123;"Pool/Statistics";2;2638800;7995;3851104;1212304;12713;4718;32768;2638800;18446744073709551615;0;2638800;7995;3851104;1212304;12713;4718;32768;2638800;0;0;0;"(none)";"Monitoring & Statistical Data"
1134;"Pool/malloc";2;3517120;3315;33663063;30145943;6005;2690;1572880;3615424;18446744073709551615;0;16384;64;16384;0;64;0;256;16384;0;0;0;"(none)";"System"
1175;"Pool/malloc/libhdbbasis.so";3;1845384;8;1845384;0;8;0;1572880;1845384;18446744073709551615;0;1845384;8;1845384;0;8;0;1572880;1845384;0;0;0;"(none)";"System"
1163;"Pool/malloc/libc.so.6";3;163832;503;26754128;26590296;991;488;98304;262136;18446744073709551615;0;163832;503;26754128;26590296;991;488;98304;262136;1;0;0;"(none)";"System"
1247;"Pool/Configurator";2;2056872;24364;3470952;1414080;29102;4738;5120;2056872;18446744073709551615;0;2003656;24245;3412168;1408512;28915;4670;1024;2003656;0;0;0;"(none)";"System"
4530;"StackAllocator";1;28573696;21;29884416;1310720;22;1;2359296;28573696;18446744073709551615;0;28573696;21;29884416;1310720;22;1;2359296;28573696;0;0;0;"(none)";"Statement Execution & Intermediate Results"
(M_HEAP_MEMORY, 12/228 rows in 911us (117 inactive, 99 invisible rows skipped))

[BUILD]  Build information: (2022-04-12 14:07:07 397 Local)
Version       : 2.00.052.00.1599235305 (fa/hana2sp05)
Build host    : lsh35314le
Build time    : 2020-09-04 18:21:41
Platform      : linuxppc64le
Compiler      : gcc (SAP release 20200227, based on SUSE gcc9-9.2.1+r275327-1.3.7) 9.2.1 20190903 [gcc-9-branch revision 275330]
Maketype      : rel
Branch        : fa/hana2sp05
Git hash      : 33d9d2f805777f29e5dafe6d33d93026bc39d78c
Git mergetime : 2020-09-04 18:01:45
Weekstone     : 0000.00.0
[OK]

[Sapienstar]

Hello again hakan.haslaman,

I guess previous info you asked was not the correct regarding the top allocators. From another trace file, I guess this is correct:

Top "M_HEAP_MEMORY" allocators (component, name, size). Ordered descending by inclusive_size_in_use.
 1: System:                                     /                                                           38.80gb (41662952113b)
 2: System:                                     Pool                                                        37.57gb (40341909738b)
 3: System:                                     Pool/PersistenceManager                                     17.83gb (19150686545b)
 4: System:                                     Pool/ColumnStore                                            13.30gb (14291338918b)
 5: Column Store Tables:                        Pool/PersistenceManager/UnifiedTableContainer               10.83gb (11638191793b)
 6: System:                                     Pool/ColumnStore/System                                     6.91gb (7427810352b)
 7: Column Store Tables:                        Pool/ColumnStore/Delta                                      5.71gb (6135117356b)
 8: System:                                     Pool/PersistenceManager/PersistentSpace                     4.02gb (4324744304b)
 9: System:                                     Pool/PersistenceManager/PersistentSpace/DefaultLPA          3.70gb (3980874368b)
10: System:                                     Pool/PersistenceManager/PersistentSpace/DefaultLPA/DataPage 3.23gb (3474235392b)
11: System:                                     Pool/PersistenceManager/VarSizeEntryFreeSpaceInformation    1.49gb (1610048576b)
12: Column Store Tables:                        Pool/PersistenceManager/UnifiedTableContainer/MVCC          1.35gb (1450569344b)
13: Monitoring & Statistical Data:              Pool/Statistics                                             1.32gb (1426556112b)
14: System:                                     Pool/malloc                                                 1.17gb (1259610631b)
15: System:                                     Pool/ResourceContainer                                      1.02gb (1101264072b)
16: System:                                     Pool/PersistenceManager/MidSizeLOBContainerFileIDMapping    1003.87mb (1052641384b)
17: Statement Execution & Intermediate Results: StackAllocator                                              840.37mb (881197056b)
18: Column Store Tables:                        Pool/ColumnStore/Main                                       682.50mb (715659418b)
19: Row Store Tables:                           Pool/RowStoreTables                                         525.97mb (551530384b)
20: Row Store Tables:                           Pool/RowStoreTables/CpbTree                                 521.37mb (546698880b)

[HakanHaslaman]

Is there a crash dump written?
If not, please check this if that matches to your issue and environment:
https://launchpad.support.sap.com/#/notes/2963432

[HakanHaslaman]

Thanks, but need the same in exclusive_size_in_use.
Also, please check the note I send before, looks similar to that information you already provide.
But not sure, if you have the same environment.

[Sapienstar]

Thank you, hakan.haslaman

Will check the Note.

Meanwhile, exclusive size.

Top "M_HEAP_MEMORY" allocators (component, name, size). Ordered descending by exclusive_size_in_use.
 1: Column Store Tables:                        Pool/PersistenceManager/UnifiedTableContainer                           9.48gb (10187622449b)
 2: System:                                     Pool/ColumnStore/System                                                 6.91gb (7427810352b)
 3: Column Store Tables:                        Pool/ColumnStore/Delta                                                  5.53gb (5942069056b)
 4: System:                                     Pool/PersistenceManager/PersistentSpace/DefaultLPA/DataPage             3.23gb (3474235392b)
 5: System:                                     Pool/PersistenceManager/VarSizeEntryFreeSpaceInformation                1.49gb (1610048576b)
 6: Column Store Tables:                        Pool/PersistenceManager/UnifiedTableContainer/MVCC                      1.35gb (1450569344b)
 7: Monitoring & Statistical Data:              Pool/Statistics                                                         1.32gb (1426556112b)
 8: Statement Execution & Intermediate Results: StackAllocator                                                          840.37mb (881197056b)
 9: System:                                     Pool/ResourceContainer                                                  672.66mb (705341624b)
10: Row Store Tables:                           Pool/RowStoreTables/CpbTree                                             521.37mb (546698880b)
11: System:                                     Pool/RowEngine/RowTableManager/MVCCManager/MVCCAllocator                517mb (542121152b) (512mb (536870912b) precharged, 64kb (65536b) in preparation)
12: Column Store Tables:                        Pool/SerializedObject                                                   481.09mb (504465483b)
13: System:                                     Pool/PersistenceManager/PersistentSpace/DefaultLPA/DataPageControlblock 472.75mb (495720960b)
14: System:                                     Pool/PersistenceManager/MidSizeLOBContainerFileIDMapping/Map            390.16mb (409118800b)
15: System:                                     Pool/malloc/libhdbcsdelta.so                                            381.60mb (400147232b)
16: System:                                     Pool/ResourceContainer/ResourceHeader                                   377.58mb (395922352b)
17: System:                                     AllocateOnlyAllocator-unlimited/FLA-UL<24592,1>/MemoryMapLevel3Nodes    307.23mb (322155200b)
18: System:                                     Pool/malloc/libhdbcsmd.so                                               221.11mb (231854376b)
19: Caches:                                     Pool/RowEngine/SQLPlan                                                  206.72mb (216765545b)
20: System:                                     Pool/PersistenceManager/MidSizeLOBContainerFileIDMapping/BackMap        194.71mb (204171776b)

[Sapienstar]

Dear Hakan,

Unfortunately, the note is not relevant as there is no DynaTrace in place.

[Sapienstar]

By the way, if anybody happens to read this, is it normal that SAP HANA stops all the services in case of OOM?

[simon25]

Hi Albin,

Yes, HANA will perform an automatic restart if the allocated memory pool (GAL) is exhausted for the tenant db or whole system including the systemdb, where there is more physical memory available on request for allocation .

If you check the "Inter process memory management" sections in the <service>.oom trace filedumps, that will show the 'Allocated' vs 'Used Bytes' for each service on the HANA landscape at the time of the oom, including the tenant indexserver nodes and systemdb services.

It's a case of checking as Hakan already described, the 'exclusive_size_in_use' for those expensive services and their top heap allocators. You should also check that the vm.map_map_count is set optimally on the system.

Troubleshooting problem heap allocators is detailed in '13. What can I do if a certain heap allocator is unusually large?' - KBA 1999997 - FAQ: SAP HANA Memory

See also SAP KBA 3011480 - How-To : Troubleshooting SAP HANA Memory Consumption

Kind Regards

Simon