Heap Utilisation is high | Hybris 6.5.0

[Former Member]

We are running into slowness of admin node after several days of server start. Usually the heap utilisation increases gradually and it consumes most of the heap space.

CMS Old gen is showing 100% consumption and it remains stagnant. Doesn't come down. Code Cache is showing 80% consumption. Par Eden Space is around 57%.

I have highlighted these things in the screen shot. Can any one please let us know the real reason for CMS Old gen to be 100%? Is it related to memory leak? We have installed App dynamics on our server and I am able to collate the above mentioned data from there.

[Former Member]

HI , Thanks for a quick turnaround. The problem at the moment is because of high heap memory consumption we are unable to take the heap dump. We get the socket error as below:

`com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file: target process not responding or HotSpot VM not loaded

Surprisingly, as shown in the below call, cmscockpit Filters are taking too long to respond (more than 20 minutes) for the following call:- /cmscockpit/zkau.

Would you be knowing why?

Thread Name:ajp-bio-8009-exec-6352 ID:168686 Time:Tue Mar 06 13:26:28 AEDT 2018 State:WAITING Priority:5 java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.zkoss.zkmax.ui.comet.CometServerPush.process(CometServerPush.java:224) org.zkoss.zkmax.ui.comet.CometProcessor.process0(CometProcessor.java:127) org.zkoss.zkmax.ui.comet.CometProcessor.process(CometProcessor.java:79) org.zkoss.zk.au.http.DHtmlUpdateServlet.doGet(DHtmlUpdateServlet.java:377) org.zkoss.zk.au.http.DHtmlUpdateServlet.doPost(DHtmlUpdateServlet.java:394) de.hybris.platform.cockpit.session.impl.HybrisDHtmlUpdateServlet.doPost(HybrisDHtmlUpdateServlet.java:48) javax.servlet.http.HttpServlet.service(HttpServlet.java:650) javax.servlet.http.HttpServlet.service(HttpServlet.java:731) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:158) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:121) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) org.springframework.security.web.access.channel.ChannelProcessingFilter.doFilter(ChannelProcessingFilter.java:157) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346) org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain$InternalFilterChain.doFilter(AbstractPlatformFilterChain.java:301) de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain$StatisticsGatewayFilter.doFilter(AbstractPlatformFilterChain.java:390) de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain$InternalFilterChain.doFilter(AbstractPlatformFilterChain.java:271) de.hybris.platform.servicelayer.web.WebAppMediaFilter.doFilter(WebAppMediaFilter.java:141) de.hybris.platform.servicelayer.web.WebAppMediaFilter$$FastClassBySpringCGLIB$$926a8da8.invoke() org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133) org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655) de.hybris.platform.servicelayer.web.WebAppMediaFilter$$EnhancerBySpringCGLIB$$66e0b21c.doFilter() de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain$InternalFilterChain.doFilter(AbstractPlatformFilterChain.java:271) de.hybris.platform.servicelayer.web.DynamicCatalogVersionActivationFilter.doFilter(DynamicCatalogVersionActivationFilter.java:90) de.hybris.platform.servicelayer.web.DynamicCatalogVersionActivationFilter$$FastClassBySpringCGLIB$$2d468805.invoke() org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133) org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655) de.hybris.platform.servicelayer.web.DynamicCatalogVersionActivationFilter$$EnhancerBySpringCGLIB$$c37de5b9.doFilter() de.hybris.platform.servicelayer.web.AbstractPlatformFil

[Former Member]

Could this be the reason of piling up the heap space?

[agrabovskis]

Hard to read the partial stack trace (next time add as an attachment), but it looks like the CMS cockpit waits for an event to be processed. I've seen the situations when due to misconfiguration of EditorArea, all the linked items are displayed (i.e. but before displaying needs to be processed via event queue). If the number of linked items is huge, then that could easily consume all the available memory.

It's weird that you can't get the thread dump - even if the heap is full, that is doable. You're missing something. Are you using jmap? (you should) check that the arguments are correct.

[agrabovskis]

It's either shortage of allocated memory or memory leak.

Take the heap dump and check what's in there (I'm using Ee MAT for that kind of tasks):

• if it is Hybris cache that's consuming the memory, then consider adjusting cache size and/or Xmx

• If it is memory leak, then the heap dump will show what to fix