ConcurrentModificationException in backoffice after 2005 upgrade

[davorin1]

Hi,

After upgrading the hybris to the 2005_4 we get seemingly random errors in the backoffice when saving an item. The issue appears only on the first save action on the itemtype with the "Please try again later, or contact your system Administrator." pop-up shown, additionally the system mages to persist the changes and afterwards everything works like usual , without the error message. The error doesn't get thrown for all the instances of the same itemtype, if I try to change some other itemtype the error gets thrown again.

Has anybody have some insights on how to fix this issue?

In the log the ConcurrentModificationException gets thrown with the following stack trace:

ERROR [hybrisHTTP17] [UiEngineImpl]
java.util.ConcurrentModificationException: null
        at java.util.HashMap$EntrySpliterator.forEachRemaining(HashMap.java:1751) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?]
        at com.hybris.cockpitng.widgets.common.explorertree.model.DefaultExplorerTreeModel.findNodesWithWrappedData(DefaultExplorerTreeModel.java:169) ~[backoffice-widgets-20.05.3-RC5.jar:?]
        at com.hybris.cockpitng.widgets.common.explorertree.model.DefaultExplorerTreeModel.findNodesByData(DefaultExplorerTreeModel.java:159) ~[backoffice-widgets-20.05.3-RC5.jar:?]
        at com.hybris.cockpitng.widgets.common.explorertree.ExplorerTreeController.findNodesToRefresh(ExplorerTreeController.java:1046) ~[backoffice-widgets-20.05.3-RC5.jar:?]
        at com.hybris.cockpitng.widgets.common.explorertree.ExplorerTreeController.onNodeRefreshRequest(ExplorerTreeController.java:1124) ~[backoffice-widgets-20.05.3-RC5.jar:?]
        at org.zkoss.zk.ui.AbstractComponent.onEvent(AbstractComponent.java:3179) ~[zk-8.6.3.1.jar:8.6.3.1]
        at org.zkoss.zk.ui.AbstractComponent.service(AbstractComponent.java:3149) ~[zk-8.6.3.1.jar:8.6.3.1]
        at org.zkoss.zk.ui.AbstractComponent.service(AbstractComponent.java:3091) ~[zk-8.6.3.1.jar:8.6.3.1]
        at org.zkoss.zk.ui.impl.EventProcessor.process(EventProcessor.java:138) ~[zk-8.6.3.1.jar:8.6.3.1]
        at org.zkoss.zk.ui.impl.UiEngineImpl.processEvent(UiEngineImpl.java:1845) ~[zk-8.6.3.1.jar:8.6.3.1]
        at org.zkoss.zk.ui.impl.UiEngineImpl.process(UiEngineImpl.java:1617) [zk-8.6.3.1.jar:8.6.3.1]
        at org.zkoss.zk.ui.impl.UiEngineImpl.execUpdate(UiEngineImpl.java:1320) [zk-8.6.3.1.jar:8.6.3.1]
        at org.zkoss.zk.au.http.DHtmlUpdateServlet.process(DHtmlUpdateServlet.java:611) [zk-8.6.3.1.jar:8.6.3.1]
        at org.zkoss.zk.au.http.DHtmlUpdateServlet.doGet(DHtmlUpdateServlet.java:487) [zk-8.6.3.1.jar:8.6.3.1]
        at org.zkoss.zk.au.http.DHtmlUpdateServlet.doPost(DHtmlUpdateServlet.java:495) [zk-8.6.3.1.jar:8.6.3.1]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:652) [servlet-api.jar:?]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:733) [servlet-api.jar:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [catalina.jar:8.5.57]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.57]
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-websocket.jar:8.5.57]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.57]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.57]
        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:209) [spring-security-web-5.2.5.RELEASE.jar:5.2.5.RELEASE]
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178) [spring-security-web-5.2.5.RELEASE.jar:5.2.5.RELEASE]
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) [spring-web-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) [spring-web-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.57]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.57]
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) [spring-web-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.57]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.57]
        at com.hybris.backoffice.mobile.filter.BackofficeMobileFilter.doFilter(BackofficeMobileFilter.java:56) [classes/:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.57]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.57]
        at de.hybris.platform.servicelayer.web.WebAppMediaFilter.doFilter(WebAppMediaFilter.java:124) [coreserver.jar:?]
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) [spring-web-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) [spring-web-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.57]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.57]
        at de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain$InternalFilterChain.doFilter(AbstractPlatformFilterChain.java:322) [coreserver.jar:?]
        at de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain$StatisticsGatewayFilter.doFilter(AbstractPlatformFilterChain.java:410) [coreserver.jar:?]
        at de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain$InternalFilterChain.doFilter(AbstractPlatformFilterChain.java:292) [coreserver.jar:?]
        at com.hybris.backoffice.security.BackofficeDynamicCatalogVersionActivationFilter.doFilter(BackofficeDynamicCatalogVersionActivationFilter.java:81) [classes/:?]
        at de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain$InternalFilterChain.doFilter(AbstractPlatformFilterChain.java:292) [coreserver.jar:?]
        at de.hybris.platform.servicelayer.web.DataSourceSwitchingFilter.doFilter(DataSourceSwitchingFilter.java:59) [coreserver.jar:?]
        at de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain$InternalFilterChain.doFilter(AbstractPlatformFilterChain.java:292) [coreserver.jar:?]
        at de.hybris.platform.servicelayer.web.session.stale.impl.DefaultStaleSessionCheckingFilterChain.doFilter(DefaultStaleSessionCheckingFilterChain.java:55) [coreserver.jar:?]
        at de.hybris.platform.servicelayer.web.SessionFilter.doFilterInternal(SessionFilter.java:122) [coreserver.jar:?]
        at de.hybris.platform.servicelayer.web.SessionFilter.doFilter(SessionFilter.java:84) [coreserver.jar:?]
        at de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain$InternalFilterChain.doFilter(AbstractPlatformFilterChain.java:292) [coreserver.jar:?]
        at de.hybris.platform.servicelayer.web.session.HybrisSpringSessionFilter.doFilter(HybrisSpringSessionFilter.java:67) [coreserver.jar:?]
        at de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain$InternalFilterChain.doFilter(AbstractPlatformFilterChain.java:292) [coreserver.jar:?]
        at com.hybris.cockpitng.modules.spring.filter.ExternalModuleContextClassLoaderFilter.doFilter(ExternalModuleContextClassLoaderFilter.java:37) [cockpit-module-aggregator-20.05.3-RC5.jar:?]
        at de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain$InternalFilterChain.doFilter(AbstractPlatformFilterChain.java:292) [coreserver.jar:?]
        at de.hybris.platform.servicelayer.web.RedirectWhenSystemIsNotInitializedFilter.doFilter(RedirectWhenSystemIsNotInitializedFilter.java:94) [coreserver.jar:?]
        at de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain$InternalFilterChain.doFilter(AbstractPlatformFilterChain.java:292) [coreserver.jar:?]
        at de.hybris.platform.servicelayer.web.TenantActivationFilter.doFilter(TenantActivationFilter.java:76) [coreserver.jar:?]
        at de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain$InternalFilterChain.doFilter(AbstractPlatformFilterChain.java:292) [coreserver.jar:?]
        at de.hybris.platform.servicelayer.web.Log4JFilter.doFilter(Log4JFilter.java:37) [coreserver.jar:?]
        at de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain$InternalFilterChain.doFilter(AbstractPlatformFilterChain.java:292) [coreserver.jar:?]
        at com.hybris.backoffice.filter.responseheaders.BackofficeResponseHeadersFilter.doFilter(BackofficeResponseHeadersFilter.java:31) [classes/:?]
        at de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain$InternalFilterChain.doFilter(AbstractPlatformFilterChain.java:292) [coreserver.jar:?]
        at de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain.processStandardFilterChain(AbstractPlatformFilterChain.java:200) [coreserver.jar:?]
        at de.hybris.platform.servicelayer.web.AbstractPlatformFilterChain.doFilterInternal(AbstractPlatformFilterChain.java:177) [coreserver.jar:?]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) [spring-web-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) [spring-web-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.57]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.57]
        at de.hybris.platform.servicelayer.web.XSSFilter.processPatternsAndDoFilter(XSSFilter.java:351) [coreserver.jar:?]
        at de.hybris.platform.servicelayer.web.XSSFilter.doFilter(XSSFilter.java:299) [coreserver.jar:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.57]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.57]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) [catalina.jar:8.5.57]
        at org.apache.catalina.core.StandardContextValve.__invoke(StandardContextValve.java:96) [catalina.jar:8.5.57]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:41002) [catalina.jar:8.5.57]
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:543) [catalina.jar:8.5.57]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) [catalina.jar:8.5.57]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [catalina.jar:8.5.57]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [catalina.jar:8.5.57]
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:690) [catalina.jar:8.5.57]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) [catalina.jar:8.5.57]
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:615) [tomcat-coyote.jar:8.5.57]
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-coyote.jar:8.5.57]
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:818) [tomcat-coyote.jar:8.5.57]
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1626) [tomcat-coyote.jar:8.5.57]
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-coyote.jar:8.5.57]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-util.jar:8.5.57]
        at java.lang.Thread.run(Thread.java:834) [?:?]

[geffchang]

What version are you upgrading from? A lower version of 2005?

[davorin1]

We were upgrading from 1808_23.

In addition, the issue persist across environments: we have cloud environments hosted on the SAP Commerce Cloud, in-house hosted test environments and it happens on my local machine.

[former_member1081436]

Hi Davorin, did you by any chance find a solution for this issue? We're facing the same issue after upgrading from 1808 to 2005.

[davorin1]

Hi Jan,
Unfortunately we didn't find a solution to this issue...
I've createad a ticket on the launchpad to get support from SAP but there's still no response.

[crescenzorega]

Hi davorin

i also have the same problem, did you find a solution?

Regards,

Crescenzo

[davorin1]

Hi crescenzorega,

Unfortunately, I don't have any update on that issue. I've created a ticket for SAP support months ago, than I've recreated it with High prio as the Production deployment is nearing by but after months of waiting for response there's still no response from the SAP support team. Makes you wander what the "Mainstream Support" and "High prio" really means on the SAP side...

Please, if you manage to find something that can help us provide an answer, I'd appreciate it much much 😄

Best regards,

Davorin

[crescenzorega]

Hi davorin ,

i tried to reset everything the backoffice

1. Press the F4 key.
2. Expand the menu in the upper right corner of the application.
3. Choose the Reset Everything button.
4. The whole configuration gets reset.
   
   Resetting Backoffice Configuration

the reset took a long, long time but it seems to have worked

I hope have been helpful.

Regards,

Crescenzo

[davorin1]

Hi @crescenzorega,

unfortunately, the backoffice is not the solution to the issue. That was the first thing that we've tried to do, at first it seemed like it did resolve the issue but after running the ant clean all and restarting the server the issue manifested itself again, it seems to be quite random and happens only on the first edit of the item type by the user. I've tried to find if any of the backoffice customizations used HashMap instead of the ConcurrentHashMap but that was not the case. Additionally, I've also tried setting the backoffice.cockpitng.reset.triggers with different parameters but that didn't solve the issue too.

P.S. Thank you for the response 😄

BR,

Davorin

[crescenzorega]

Hi davorin ,

if you tried to redefine the findNodesWithWrappedData method

protected List<NavigationNode> findNodesWithWrappedData(final Object data)
{
	return nodeDataCache.entrySet().stream().filter(entry -> entry.getKey() instanceof ObjectWithComponentContext)
			.filter(entry -> ((ObjectWithComponentContext) entry.getKey()).getInputObject().equals(data)).map(Map.Entry::getValue)
			.flatMap(Collection::stream).collect(Collectors.toList());
}

not using the stream but the old approach of for loops and if and map ?

[davorin1 · Accepted solution]

Solved the issue by disabling the Backoffice label cache warmup by setting the following property in the local.properties:

backoffice.cockpitng.explorertree.labelcache.warmup.enabled=false