High memory usage by Pool/malloc/libhdbrskernel.so

[svkarthikeyantce]

Hi Experts,

In our HANA system, I could see the memory usage by Pool/malloc/libhdbrskernel.so is very high that it is utilizing almost 45% of licensed memory. When checked with hdbcons command, below seems to be top consumer of the memory.

> mm bl -rt Pool/malloc/libhdbrskernel.so ## Start command at: 2021-08-17 10:52:04.564 854528935484b (1 block) in use at: Dumping saved stack trace 9223512740347585906, 1 frames: 1: 0x00007ff815b52572 in TrexService::RemoteService::fgetlines(ltt_adp::basic_string<char, ltt::char_traits<char>, ltt::integral_constant<bool, true> > const&, unsigned long, long, ltt_adp::basic_string<char, ltt::char_traits<char>, ltt::integral_constant<bool, true> > const&, unsigned int, long*, unsigned long*, ltt::basic_string<char, ltt::char_traits<char>, ltt::integral_constant<int, 64> >*)+0xbf0 at memory.hpp:1202 (libhdbbasement.so)

No long running sql's are there in the system. Cause is not evident with logs available. Can you help in identifying the root cause of this issue.

Regards, Karthikeyan.SV

[dvankempen]

Hi Karthikeyan,

For this type of issues, probably best to create an incident and have SAP Support look into the issue.

[svkarthikeyantce]

I have raised this case with SAP.

Meanwhile we rebooted HANA still the heap memory allocation grew to 850 GB. On thread analysis, I could see many threads are running with THREAD_METHOD=ExpensiveStatementTraceLoader with state "Network Poll" and it has a Statement_Execution_ID associated with it, which I couldn't find in Expensive/Executed statements.

Moreover, no changes made to expensive statement trace and no filter applied with it.

Any pointers on this analysis is much appreciated.

[svkarthikeyantce]

I checked these notes earlier but couldn't find the exact issue. Temporary solution proposed may work, but that needs to applied during every revision update. Also no long running SQL's are there in the system.

HANA version that we use is of 2.0 SP05 Rev. 54

[SriKrishna]

hi,

Please refer SAP Note 1999997 - FAQ: SAP HANA Memory

2275252 - Memory Leak in "Pool/malloc/libhdbrskernel.so" When SqlScript Procedures or UDFs Get Compiled

2253121 - Memory Leak in Pool Pool/malloc/libhdbrskernel.so when Using Batch Inserts Into Rowstore Tables