SAP ECC6 memory and paging issues

[#1]

Dear Experts

I have recently upgraded my 4.6C systems to an ECC 6 system (DB2 LUW 9.5 on AIX 5.3 TL9 64 Bit OS)

I have been running the LPAR with 14 GB of memory and we are around 100-200+ users using the system, I was monitoring using nmon and found that Physical Memory was around 99.8% Used (14311.8MB and 22.6MB was free) also the paging space was around 37.2% in result causing the system at times to run slow which can have a very negative effect on the users.

After further investigation I found that after a system restart the Physical Memory would start around 50.9% and increased at a steady pace until it reached 99.8% that is when the system would start using the paging space which would steadily increase, I found that the only solution was a system restart at least once a week to reduce the memory consumption.

At first glance it looked like a database manger memory leak with the process db2sysc, so I searched the net with the search words u201Cdb2 memory leaku201D and found the following APARs and notes.

APAR JR30285 - Pervasive memory leak when compiling SQL statements that use SQL/XML functions

APAR IZ35230 - There is a pervasive unix-specific private memory leak in the security component

Note 1288341 - Memory leak in APPLHEAPSZ -> SQL0954C

Note 1352361 - Memory leak in shared memory area abrfci

Note 1147821 - DB6: Known Errors and available Fixes in DB2 9.5 LUW

After reading the notes and APARs I decided to updated DB2 to the latest fix pack (5SAP), but after the fix pack was implemented it did not solve the memory problem

I started look at different problems with SAP ECC6, db2 and AIX with paging/memory problems and I found the following notes to do with AIX memory and paging but none of them helped as all parameters and settings were set accordingly

789477 - Large extended memory on AIX (64-bit) as of Kernel 6.20

191801 - AIX 64-bit with very large amount of Extended Memory

973227 - AIX Virtual Memory Management: Tuning Recommendations

884393 - AIX saposcol consumes large amount of memory.

856848 u2013 AIX Extended Memory Disclaiming

1048686 u2013 Recommended AIX settings for SAP

1121904 u2013 SAP on AIX: Recommendations for Paging

1086130 u2013 DB6: DB2 Standard Parameter Settings

After even more investigation I found the following evidence suggesting AIX Virtual Memory Manager might have a problem

[Former Member]

I found reports from nmon analyzer that are reported on daily and are kept for over 6 months, so I have compiled a spread sheet with the relevant information about memory usage and it starts at a system restart then collected for 7 days, with before and after the upgrade, I did a comparison and below is what I found:

nmon analyzer results:

MEMNEW:

In the before spread sheet for the MEMNEW worksheets show that the average process usage is around 70%-100% and occasionally spiking but it always returning to around 70%, and in the after spread sheet for the MEMNEW worksheets its shown that the average process usage is around 90%-100%.

MEMUSE:

In the before spread sheet for the MEMUSE worksheets show that the %numperm is rarely below the %minperm and the %comp is never above the %maxperm and in the after spread sheet for the MEMUSE worksheets show that the %numperm is always below the %minperm and the %comp is always above the %maxperm thus the virtual memory manager is forced to page anything at will

If any one would like to view the spread sheets I can e-mail them the following: nmon_P1LPAR03_Before.xls and nmon_P1LPAR03_After.xls

At this point I have hit a wall and donu2019t know what I can do to fix the problem

If any one can please help me it would be greatly appreciated

Regards,

Simon Grzebieta

DB2 DBA Consultant

[Former Member]

Hi Simon, also running DB2 9.5 with FP5sap on AIX 5.3 TL11 and ECC6 (unicode), upgraded from 4.6c late last year and seem to have similar situation to what you describe -> memory in use by DB2 instance grows over time, pushing up swap utilisation, only way to resolve is to restart DB2 instance (& SAP)

Wondering if you use STMM? Do you have instance_memory as automatic or a fixed value?

Daniel Parker

[Former Member]

>

> Wondering if you use STMM? Do you have instance_memory as automatic or a fixed value?

STMM is turned off and db2 instance_memory is set to a fixed value of 1000000 4KB pages or 3.8 GB