SAP ECC6 memory and paging issues

[#1]

Dear Experts

I have recently upgraded my 4.6C systems to an ECC 6 system (DB2 LUW 9.5 on AIX 5.3 TL9 64 Bit OS)

I have been running the LPAR with 14 GB of memory and we are around 100-200+ users using the system, I was monitoring using nmon and found that Physical Memory was around 99.8% Used (14311.8MB and 22.6MB was free) also the paging space was around 37.2% in result causing the system at times to run slow which can have a very negative effect on the users.

After further investigation I found that after a system restart the Physical Memory would start around 50.9% and increased at a steady pace until it reached 99.8% that is when the system would start using the paging space which would steadily increase, I found that the only solution was a system restart at least once a week to reduce the memory consumption.

At first glance it looked like a database manger memory leak with the process db2sysc, so I searched the net with the search words u201Cdb2 memory leaku201D and found the following APARs and notes.

APAR JR30285 - Pervasive memory leak when compiling SQL statements that use SQL/XML functions

APAR IZ35230 - There is a pervasive unix-specific private memory leak in the security component

Note 1288341 - Memory leak in APPLHEAPSZ -> SQL0954C

Note 1352361 - Memory leak in shared memory area abrfci

Note 1147821 - DB6: Known Errors and available Fixes in DB2 9.5 LUW

After reading the notes and APARs I decided to updated DB2 to the latest fix pack (5SAP), but after the fix pack was implemented it did not solve the memory problem

I started look at different problems with SAP ECC6, db2 and AIX with paging/memory problems and I found the following notes to do with AIX memory and paging but none of them helped as all parameters and settings were set accordingly

789477 - Large extended memory on AIX (64-bit) as of Kernel 6.20

191801 - AIX 64-bit with very large amount of Extended Memory

973227 - AIX Virtual Memory Management: Tuning Recommendations

884393 - AIX saposcol consumes large amount of memory.

856848 u2013 AIX Extended Memory Disclaiming

1048686 u2013 Recommended AIX settings for SAP

1121904 u2013 SAP on AIX: Recommendations for Paging

1086130 u2013 DB6: DB2 Standard Parameter Settings

After even more investigation I found the following evidence suggesting AIX Virtual Memory Manager might have a problem

[Former Member]

Hi All

Just a quick update,

It looks like my problem is solved, I recently received the results from the SAP GoingLive functional upgrade check verification session, and SAP had suggested specific SAP and DB2 parameter changes (Listed Below) which were changed over the weekend and after monitoring the systems for the past 4 days it seems that the changes have fixed the memory and paging issue I have been experiencing.

I believe that the the setting the database memory, instance memory and the page cleaner trigger were ultimately responsible for the memory and paging issues. But I will be conducting some extra test to try and work out which parameters were directly responsible for the memory and paging problems

DB2 Database Configuration Changes:

Database Memory (database_memory) from: AUTOMATIC to: COMPUTED

Utilities heap size (util_heap_sz) from: 10000 to: 50000

Threshold for page cleaner trigger (chngpgs_thresh) from: 40 to: 20

DB2 Database Manager Configuration Changes:

Keep fenced process (keepfenced) from: NO to: YES

Number of fenced processes (fenced_pool Maximum) from: 5 to: 10

SAP Instance Parameter Changes:

Number of nametab entries administrated (rsdb/ntab/entrycount) from: 50000 to: 65000

Size of generic table buffer (zcsa/table_buffer_area) from: 129850000 to: 168805000

Size of short-term memory (em/short_area) from: 1600000 to: 2000000

Size of short-term memory (ztta/short_area) from: 1600000 to: 2000000

AIX: Segs Implementation Version (ES/SHM_SEGS_VERSION) from: N/A to: 2

Max. number of internal asynchronous messages (rdisp/max_arq) from: 500 to: 2000

Network Connection Keep Alive Timeout (icm/keep_alive_timeout) from: 60 to: 90

I will post another update after I have worked out which parameters a directly responsible

I hope this helps Regards

Simon Grzebieta

DB2 DBA Consultant