data hub publication stuck in pending status V6.2

[Former Member]

We have many pools created in Datahub. This is the flow when we have a pool stuck in publication PENDING. I found that if OOB clean up properties are all turned off (all set to false), this error no longer happens. But this is not a good solution as there's major performance impact.

1. all pools publishing fine

2. Pool X has a new publication in failure status.

3. Check error in publication ID and nothing found (datahub-webapp/v1/target-system-publications/{publication ID}/errors?pageNumber=0&pageSize=500)

4. All new publication triggered in pool X has publication status = PENDING

Note: this is different from FAILURE publication in this flow:

1. All pool publication are in failure status.

2. Check error in publication, found error on authentication to hybris (datahub-webapp/v1/target-system-publications/{publication ID}/errors?pageNumber=0&pageSize=500)

3. fix credential errors

4. New publication triggered are SUCCESS

This is in API /datahub-webapp/v1/pools/{pool}/publications/ We know the pool is in publication started status, but the pending publication is never started (or turn into in progress). The feed is stuck then.

I tried POST to trigger publication in the pool /datahub-webapp/v1/pools/{pool name}/publications, it creates a new publication but also in PENDING status.

 <publicationActionData>
 <actionId>30</actionId>
 <type>PUBLICATION</type>
 <status>PENDING</status>
 <poolName>JDEProductPool</poolName>
 <targetSystemPublications>
 <publicationId>113</publicationId>
 <status>PENDING</status>
 <targetSystemName>HybrisProduct</targetSystemName>
 <actionId>30</actionId>
 <poolName>JDEProductPool</poolName>
 <canonicalItemCount>0</canonicalItemCount>
 <internalErrorCount>0</internalErrorCount>
 <externalErrorCount>0</externalErrorCount>
 <ignoredCount>0</ignoredCount>
 </targetSystemPublications>
 </publicationActionData>
 <publicationActionData>
 <actionId>27</actionId>
 <type>PUBLICATION</type>
 <startTime>2017-02-17T14:06:08Z</startTime>
 <endTime>2017-02-17T14:06:31Z</endTime>
 <status>FAILURE</status>
 <poolName>JDEProductPool</poolName>
 <targetSystemPublications>
 <publicationId>101</publicationId>
 <startTime>2017-02-17T14:06:08Z</startTime>
 <endTime>2017-02-17T14:06:31Z</endTime>
 <status>FAILURE</status>
 <targetSystemName>HybrisProduct</targetSystemName>
 <actionId>27</actionId>
 <poolName>JDEProductPool</poolName>
 <canonicalItemCount>6</canonicalItemCount>
 <internalErrorCount>0</internalErrorCount>
 <externalErrorCount>2</externalErrorCount>
 <ignoredCount>0</ignoredCount>
 </targetSystemPublications>
 </publicationActionData>

I see these errors in DH log, not sure if they are relevant or not.

 2017-02-10 14:32:15,541 [ERROR] [c.h.d.s.i.DefaultDataHubEventListenerRegistry] com.hybris.datahub.service.jmx.TargetSystemPublicationCompletionJmxEventHandler threw an exception on TargetSystemPublicationCompletedEvent.  Continuing to notify other listeners.  org.springframework.orm.jpa.JpaObjectRetrievalFailureException: Unable to find com.hybris.datahub.domain.jpa.entities.ManagedCanonicalItemEntity with id 5580; nested exception is javax.persistence.EntityNotFoundException: Unable to find com.hybris.datahub.domain.jpa.entities.ManagedCanonicalItemEntity with id 5580
 

and

 java.lang.IllegalStateException: Target system publication with id 86 has already been completed or is pending. Aborting further completion actions.
 handle publication completed event2017-02-10 14:32:15,848 [INFO] [c.a.d.e.f.p.PublicationCompletedEventListener] Handling publication completed event for pool : InvoicePool
  at java.lang.Thread.run(Thread.java:745) [na:1.8.0_92]
  at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-coyote.jar:7.0.65]
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_92]
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_92]
  at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318) [tomcat-coyote.jar:7.0.65]
  at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625) [tomcat-coyote.jar:7.0.65]
  at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1079) [tomcat-coyote.jar:7.0.65]
  at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:423) [catalina.jar:7.0.65]
  at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) [catalina.jar:7.0.65]
  at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:956) [catalina.jar:7.0.65]
  at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) [catalina.jar:7.0.65]
  at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170) [catalina.jar:7.0.65]
  at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505) [catalina.jar:7.0.65]
  at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) [catalina.jar:7.0.65]
  at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) [catalina.jar:7.0.65]
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.65]
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.65]
  at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262) [spring-web-4.2.7.RELEASE.jar:4.2.7.RELEASE]
  at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346) [spring-web-4.2.7.RELEASE.jar:4.2.7.RELEASE]
  at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
  at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
  at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.7.RELEASE.jar:4.2.7.RELEASE]
  at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.7.RELEASE.jar:4.2.7.RELEASE]
  at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.7.RELEASE.jar:4.2.7.RELEASE]
  at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-4.2.7.RELEASE.jar:4.2.7.RELEASE]
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.7.RELEASE.jar:4.2.7.RELEASE]
  at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:215) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
  at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
  at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
  at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
  at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
  at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
  at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
  at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) [spring-security-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.65]
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.65]
  at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat7-websocket.jar:7.0.65]
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.65]
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) [catalina.jar:7.0.65]
  at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:223) [jersey-container-servlet-core-2.22.1.jar:na]
  at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:336) [jersey-container-servlet-core-2.22.1.jar:na]
  at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:383) [jersey-container-servlet-core-2.22.1.jar:na]
  at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:425) [jersey-container-servlet-core-2.22.1.jar:na]
  at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:471) [jersey-container-servlet-core-2.22.1.jar:na]
  at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [jersey-server-2.22.1.jar:na]
  at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [jersey-server-2.22.1.jar:na]
  at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [jersey-common-2.22.1.jar:na]
  at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.22.1.jar:na]
  at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.22.1.jar:na]
  at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.22.1.jar:na]
  at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.22.1.jar:na]
  at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.22.1.jar:na]
  at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) ~[jersey-server-2.22.1.jar:na]
  at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[jersey-server-2.22.1.jar:na]
  at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[jersey-server-2.22.1.jar:na]
  at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[jersey-server-2.22.1.jar:na]
  at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[jersey-server-2.22.1.jar:na]
  at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160) ~[jersey-server-2.22.1.jar:na]
  at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[jersey-server-2.22.1.jar:na]
  at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[jersey-server-2.22.1.jar:na]
  at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[jersey-server-2.22.1.jar:na]
  at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_92]
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_92]
  at sun.reflect.GeneratedMethodAccessor501.invoke(Unknown Source) ~[na:na]
  at com.hybris.datahub.rest.resources.CoreAdapterResource$$EnhancerBySpringCGLIB$$8e9fcfa1.completePublication(<generated>) ~[datahub-core-export-service-6.2.0.0-RC4.jar:6.2.0.0-RC4]
  at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655) ~[spring-aop-4.2.7.RELEASE.jar:4.2.7.RELEASE]
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.2.7.RELEASE.jar:4.2.7.RELEASE]
  at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.2.7.RELEASE.jar:4.2.7.RELEASE]
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:168) ~[spring-aop-4.2.7.RELEASE.jar:4.2.7.RELEASE]
  at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52) ~[spring-aop-4.2.7.RELEASE.jar:4.2.7.RELEASE]
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.2.7.RELEASE.jar:4.2.7.RELEASE]
  at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720) ~[spring-aop-4.2.7.RELEASE.jar:4.2.7.RELEASE]
  at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.2.7.RELEASE.jar:4.2.7.RELEASE]
  at com.hybris.datahub.rest.resources.CoreAdapterResource$$FastClassBySpringCGLIB$$b3756934.invoke(<generated>) ~[datahub-core-export-service-6.2.0.0-RC4.jar:6.2.0.0-RC4]
  at com.hybris.datahub.rest.resources.CoreAdapterResource.completePublication(CoreAdapterResource.java:92) ~[datahub-core-export-service-6.2.0.0-RC4.jar:6.2.0.0-RC4]
  at com.sun.proxy.$Proxy189.completePublication(Unknown Source) ~[na:na]
  at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) ~[spring-aop-4.2.7.RELEASE.jar:4.2.7.RELEASE]
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.2.7.RELEASE.jar:4.2.7.RELEASE]
  at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) ~[spring-tx-4.2.6.RELEASE.jar:4.2.6.RELEASE]
  at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281) ~[spring-tx-4.2.6.RELEASE.jar:4.2.6.RELEASE]
  at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99) ~[spring-tx-4.2.6.RELEASE.jar:4.2.6.RELEASE]
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.2.7.RELEASE.jar:4.2.7.RELEASE]
  at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) ~[spring-aop-4.2.7.RELEASE.jar:4.2.7.RELEASE]
  at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302) ~[spring-aop-4.2.7.RELEASE.jar:4.2.7.RELEASE]
  at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_92]
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_92]
  at sun.reflect.GeneratedMethodAccessor503.invoke(Unknown Source) ~[na:na]
  at com.hybris.datahub.core.adapter.facade.DefaultCoreAdapterFacade.completePublication(DefaultCoreAdapterFacade.java:76) ~[datahub-core-export-service-6.2.0.0-RC4.jar:na]
  at com.hybris.datahub.service.impl.DefaultPublicationActionService.completeTargetSystemPublication(DefaultPublicationActionService.java:53) ~[datahub-service-6.2.0.0-RC4.jar:6.2.0.0-RC4]
  at com.sun.proxy.$Proxy137.completeTargetSystemPublication(Unknown Source) ~[na:na]
  at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) ~[spring-aop-4.2.7.RELEASE.jar:4.2.7.RELEASE]
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.2.7.RELEASE.jar:4.2.7.RELEASE]
  at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) ~[spring-tx-4.2.6.RELEASE.jar:4.2.6.RELEASE]
  at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281) ~[spring-tx-4.2.6.RELEASE.jar:4.2.6.RELEASE]
  at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99) ~[spring-tx-4.2.6.RELEASE.jar:4.2.6.RELEASE]
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.2.7.RELEASE.jar:4.2.7.RELEASE]
  at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) ~[spring-aop-4.2.7.RELEASE.jar:4.2.7.RELEASE]
  at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302) ~[spring-aop-4.2.7.RELEASE.jar:4.2.7.RELEASE]
  at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_92]
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_92]
  at sun.reflect.GeneratedMethodAccessor504.invoke(Unknown Source) ~[na:na]
  at com.hybris.datahub.service.impl.DefaultTargetSystemPublicationCompletionService.completeTargetSystemPublication(DefaultTargetSystemPublicationCompletionService.java:95) ~[datahub-service-6.2.0.0-RC4.jar:6.2.0.0-RC4]
 java.lang.IllegalStateException: Target system publication with id 84 has already been completed or is pending. Aborting further completion actions.
  ... 33 common frames omitted
  at org.springframework.orm.jpa.JpaTransactionManager.doCommit(JpaTransactionManager.java:517) ~[spring-orm-4.2.7.RELEASE.jar:4.2.7.RELEASE]
  at org.hibernate.jpa.internal.TransactionImpl.commit(TransactionImpl.java:74) ~[hibernate-entitymanager-4.3.6.Final.jar:4.3.6.Final]
 Caused by: javax.persistence.RollbackException: Transaction marked as rollbackOnly
  at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [scala-library-2.11.5.jar:na]
  at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [scala-library-2.11.5.jar:na]
  at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [scala-library-2.11.5.jar:na]
  at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [scala-library-2.11.5.jar:na]
  at akka.dispatch.Mailbox.exec(Mailbox.scala:234) [akka-actor_2.11-2.4.8.jar:na]
  at akka.dispatch.Mailbox.run(Mailbox.scala:224) [akka-actor_2.11-2.4.8.jar:na]
  at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) [akka-actor_2.11-2.4.8.jar:na]
  at akka.actor.ActorCell.invoke(ActorCell.scala:495) [akka-actor_2.11-2.4.8.jar:na]
  at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) [akka-actor_2.11-2.4.8.jar:na]
  at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:47) ~[akka-actor_2.11-2.4.8.jar:na]
  at akka.actor.Actor$class.aroundReceive(Actor.scala:484) ~[akka-actor_2.11-2.4.8.jar:na]
  at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) ~[scala-library-2.11.5.jar:na]
  at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) ~[akka-actor_2.11-2.4.8.jar:na]
  at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) ~[scala-library-2.11.5.jar:na]
  at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) ~[akka-actor_2.11-2.4.8.jar:na]
  at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) ~[akka-actor_2.11-2.4.8.jar:na]
  at com.hybris.datahub.akka.event.PoolEventHandler.handleDeliverPoolEvent(PoolEventHandler.java:69) ~[datahub-service-6.2.0.0-RC4.jar:6.2.0.0-RC4]
  at com.sun.proxy.$Proxy143.notifyListeners(Unknown Source) ~[na:na]
  at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) ~[spring-aop-4.2.7.RELEASE.jar:4.2.7.RELEASE]
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.2.7.RELEASE.jar:4.2.7.RELEASE]
  at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) ~[spring-tx-4.2.6.RELEASE.jar:4.2.6.RELEASE]
  at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281) ~[spring-tx-4.2.6.RELEASE.jar:4.2.6.RELEASE]
  at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99) ~[spring-tx-4.2.6.RELEASE.jar:4.2.6.RELEASE]
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.2.7.RELEASE.jar:4.2.7.RELEASE]
  at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) ~[spring-aop-4.2.7.RELEASE.jar:4.2.7.RELEASE]
  at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302) ~[spring-aop-4.2.7.RELEASE.jar:4.2.7.RELEASE]
  at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_92]
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_92]
  at sun.reflect.GeneratedMethodAccessor327.invoke(Unknown Source) ~[na:na]
  at com.hybris.datahub.service.impl.DefaultDataHubEventListenerRegistry.notifyListeners(DefaultDataHubEventListenerRegistry.java:64) ~[datahub-service-6.2.0.0-RC4.jar:6.2.0.0-RC4]
  at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:150) ~[spring-tx-4.2.6.RELEASE.jar:4.2.6.RELEASE]
  at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:730) ~[spring-tx-4.2.6.RELEASE.jar:4.2.6.RELEASE]
  at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:761) ~[spring-tx-4.2.6.RELEASE.jar:4.2.6.RELEASE]
  at org.springframework.orm.jpa.JpaTransactionManager.doCommit(JpaTransactionManager.java:526) ~[spring-orm-4.2.7.RELEASE.jar:4.2.7.RELEASE]
 org.springframework.transaction.TransactionSystemException: Could not commit JPA transaction; nested exception is javax.persistence.RollbackException: Transaction marked as rollbackOnly
 

[cieslo · Accepted solution]

Hi,

If you are not using version 6.2.0.4 you should upgrade to this version and provide datahub.cleanup.publisheditems.time.delay property in DataHub's local.properties file. The value should be calculated in the same way as for datahub.cleanup.canonicalitem.time.delay property. This is time in seconds for the longest expected running publication + time for post processing + some margin. Default values in 6.2.0.4 are 12 hours (43200 seconds) for both of the properties.

[Former Member · Accepted solution]

Hi Marcin, is this new in 6.2.0.4? I'm running 6.2.0.0, is this feature not in 6.2.0.0?

[cieslo · Accepted solution]

Hi,

datahub.cleanup.publisheditems.time.delay is not present in 6.2.0.0. You should use 6.2.0.4 version.

[Former Member · Accepted solution]

Hi Marcin, when I upgraded to 6.2.0.4, we are facing this issue: error: https://answers.sap.com/questions/12762171/view.html

com.hybris.datahub.conversion.base.ConversionException: Failed to convert from com.hybris.datahub.domain.jpa.entities.ManagedRawItemEntity@233d to RawItem

This happens when we turn on the clean up properties. The pool is working fine for a while, then a composition has failure status, and all subsequent compositions in that pool has failure status.

[cieslo · Accepted solution]

Hi,

Can you attach logs from the moment of start till the exception is thrown?

[Slava]

Richard,

it's hard to say what exactly is going on without knowing the details of the deployments and actions performed on the DataHub.

Seems that first stack trace may be connected to data cleaning activity, when canonical items were removed by some process before the publication has finished.

Second stack trace possible indicates that the publication has been finished (perhaps manually) then datahub adapter reported the real publication completion.

Again, not sure about the scenario that is possible involved here but, if you're asking about publications stuck in PENDING status, then my answer to this question https://answers.sap.com/questions/12761702/datahub-target-system-publication-items-gets-stuck.html?c... may help.

[Former Member]

Hi Slava, updated the question -

The previous publication complete with failure (do not know why. there's nothing in /datahub-webapp/v1/target-system-publications/101/errors?pageNumber=0&pageSize=500)

But even the, the pending publication does not start. Any thoughts?

[Slava]

Check carefully publication statuses in the pools. 1) DataHub does not start new publication in the same pool when there is already one in IN_PROGRESS status. However, it can start publications in a different pool, in which there are no currently running publications. So, all new publications will stay in PENDING status until that IN_PROGRESS publication will turn into SUCCESS, COMPLETE_W_ERROR or FAILURE. 2) Errors may not exists for FAILURE status. If, for example, DataHub has failed even connecting to the target system, the status will be FAILURE and there will be no item level errors.

[Former Member]

Hi Slava, the problem is exactly that in the same pool, the PENDING publication is stuck in PENDING, while the previous publication is in FAILURE.

At the same time we know this is not because of connectivity because other pools are publishing fine and the broken pool were also working fine for a few publications.

[Slava]

Richard,

this is definitely not an expected behavior. I would suggest to file a bug against DataHub. List exact version you use, all extensions deployed with DataHub, provide step-by-step instructions to reproduce the problem (attach data sets used in the steps).