DataHub update extensions error

[Former Member]

Hello experts, we are implementing some custom extensions in datahub.

Time to timw we do change the extensions. But when we stgain the datahub we get errors during startup due to loading extensions metadata to database and the only way we have found to solve them is restarting datahub with parameter datahub.autoInitMode=create-drop This solve our problema but this is not usefull in our productive system. Is there any way to avoid use of datahub.autoInitMode=create-drop to solve our problema? Thanks io

[Former Member]

Many thanks Slava.

We have tried with ignore option and sometimes it goes ok , but sometimes we get errors:

2018-03-08 12:13:15,476 [ERROR] [c.h.d.e.l.i.DefaultExtensionLoaderService] Could not load extension saparticle com.hybris.datahub.validation.ValidationException: [ValidationFailure{FATAL, rawAttribute: "Canonical attribute transformation cannot be modifiedCanonicalVariantArticle, articleID, RawARTMAS"}] at com.hybris.datahub.validation.AbstractValidator.validate(AbstractValidator.java:32) ~[datahub-service-6.5.0.4-RC2.jar:6.5.0.4-RC2] at com.hybris.datahub.facade.metadata.DefaultTransformationItemClassFacade.createNewAttribute(DefaultTransformationItemClassFacade.java:42) ~[datahub-service-6.5.0.4-RC2.jar:6.5.0.4-RC2] at com.hybris.datahub.facade.metadata.DefaultTransformationItemClassFacade.createNewAttribute(DefaultTransformationItemClassFacade.java:32) ~[datahub-service-6.5.0.4-RC2.jar:6.5.0.4-RC2] at com.hybris.datahub.facade.metadata.AbstractItemClassFacade.createAttribute(AbstractItemClassFacade.java:36) ~[datahub-service-6.5.0.4-RC2.jar:6.5.0.4-RC2] at sun.reflect.GeneratedMethodAccessor229.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_45] at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_45] at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) ~[spring-aop-4.3.3.RELEASE.jar:4.3.3.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) ~[spring-aop-4.3.3.RELEASE.jar:4.3.3.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.3.3.RELEASE.jar:4.3.3.RELEASE] at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99) ~[spring-tx-4.3.3.RELEASE.jar:4.3.3.RELEASE] at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281) ~[spring-tx-4.3.3.RELEASE.jar:4.3.3.RELEASE] at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) ~[spring-tx-4.3.3.RELEASE.jar:4.3.3.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.3.RELEASE.jar:4.3.3.RELEASE] at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) ~[spring-aop-4.3.3.RELEASE.jar:4.3.3.RELEASE] at com.sun.proxy.$Proxy178.createAttribute(Unknown Source) ~[na:na] at com.hybris.datahub.extension.load.impl.CanonicalModelLoader.loadAttributeTransformations(CanonicalModelLoader.java:104) ~[datahub-extension-loader-6.5.0.4-RC2.jar:6.5.0.4-RC2] at com.hybris.datahub.extension.load.impl.CanonicalModelLoader.createCanonicalAttribute(CanonicalModelLoader.java:90) ~[datahub-extension-loader-6.5.0.4-RC2.jar:6.5.0.4-RC2] at com.hybris.datahub.extension.load.impl.CanonicalModelLoader.lambda$loadCanonicalItemType$0(CanonicalModelLoader.java:59) ~[datahub-extension-loader-6.5.0.4-RC2.jar:6.5.0.4-RC2] at com.hybris.datahub.extension.load.impl.CanonicalModelLoader$$Lambda$169/900212376.accept(Unknown Source) ~[na:na] at java.util.ArrayList.forEach(ArrayList.java:1249) ~[na:1.8.0_45] at com.hybris.datahub.extension.load.impl.CanonicalModelLoader.loadCanonicalItemType(CanonicalModelLoader.java:59) ~[datahub-extension-loader-6.5.0.4-RC2.jar:6.5.0.4-RC2] at com.hybris.datahub.extension.load.impl.CanonicalModelLoader$$Lambda$168/1594293614.accept(Unknown Source) ~[na:na] at java.util.ArrayList.forEach(ArrayList.java:1249) ~[na:1.8.0_45] at com.hybris.datahub.extension.load.impl.CanonicalModelLoader.loadCanonicalItems(CanonicalModelLoader.java:46) ~[datahub-extension-loader-6.5.0.4-RC2.jar:6.5.0.4-RC2] at com.hybris.datahub.extension.load.impl.DefaultExtensionImportService.loadExtension(DefaultExtensionImportService.java:44) ~[datahub-extension-loader-6.5.0.4-RC2.jar:6.5.0.4-RC2] at sun.reflect.GeneratedMethodAccessor235.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_45] at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_45] at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) ~[spring-aop-4.3.3.RELEASE.jar:4.3.3.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) ~[spring-aop-4.3.3.RELEASE.jar:4.3.3.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.3.3.RELEASE.jar:4.3.3.RELEASE] at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99) ~[spring-tx-4.3.3.RELEASE.jar:4.3.3.RELEASE] at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281) ~[spring-tx-4.3.3.RELEASE.jar:4.3.3.RELEASE] at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) ~[spring-tx-4.3.3.RELEASE.jar:4.3.3.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.3.RELEASE.jar:4.3.3.RELEASE] at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) ~[spring-aop-4.3.3.RELEASE.jar:4.3.3.RELEASE] at com.sun.proxy.$Proxy182.loadExtension(Unknown Source) ~[na:na] at com.hybris.datahub.extension.load.impl.DefaultExtensionLoaderService.loadExtension(DefaultExtensionLoaderService.java:86) [datahub-extension-loader-6.5.0.4-RC2.jar:6.5.0.4-RC2] at com.hybris.datahub.extension.load.impl.DefaultExtensionLoaderService$$Lambda$167/1850428445.accept(Unknown Source) [datahub-extension-loader-6.5.0.4-RC2.jar:6.5.0.4-RC2] at java.util.Iterator.forEachRemaining(Iterator.java:116) [na:1.8.0_45] at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) [na:1.8.0_45] at java.util.stream.ReferencePipeline$Head.forEachOrdered(ReferencePipeline.java:590) [na:1.8.0_45] at com.hybris.datahub.extension.load.impl.DefaultExtensionLoaderService.loadExtensionsInOrder(DefaultExtensionLoaderService.java:78) [datahub-extension-loader-6.5.0.4-RC2.jar:6.5.0.4-RC2] at com.hybris.datahub.extension.load.impl.DefaultExtensionLoaderService.loadExtensions(DefaultExtensionLoaderService.java:71) [datahub-extension-loader-6.5.0.4-RC2.jar:6.5.0.4-RC2] at com.hybris.datahub.extension.load.impl.DefaultExtensionLoaderService.loadExtensionsFromClasspath(DefaultExtensionLoaderService.java:44) [datahub-extension-loader-6.5.0.4-RC2.jar:6.5.0.4-RC2] at com.hybris.datahub.spring.ContextRefreshedEventListener.onApplicationEvent(ContextRefreshedEventListener.java:43) [classes/:na] at com.hybris.datahub.spring.ContextRefreshedEventListener.onApplicationEvent(ContextRefreshedEventListener.java:25) [classes/:na] at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:166) [spring-context-4.3.3.RELEASE.jar:4.3.3.RELEASE] at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:138) [spring-context-4.3.3.RELEASE.jar:4.3.3.RELEASE] at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:382) [spring-context-4.3.3.RELEASE.jar:4.3.3.RELEASE] at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:336) [spring-context-4.3.3.RELEASE.jar:4.3.3.RELEASE] at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:877) [spring-context-4.3.3.RELEASE.jar:4.3.3.RELEASE] at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:544) [spring-context-4.3.3.RELEASE.jar:4.3.3.RELEASE] at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:444) [spring-web-4.3.3.RELEASE.jar:4.3.3.RELEASE] at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:326) [spring-web-4.3.3.RELEASE.jar:4.3.3.RELEASE] at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107) [spring-web-4.3.3.RELEASE.jar:4.3.3.RELEASE] at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5118) [catalina.jar:7.0.72] at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5634) [catalina.jar:7.0.72] at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) [catalina.jar:7.0.72] at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) [catalina.jar:7.0.72] at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:875) [catalina.jar:7.0.72] at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) [catalina.jar:7.0.72] at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) [catalina.jar:7.0.72] at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) [catalina.jar:7.0.72] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_45] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_45] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]

This happens when, for instance, we add some custom extensions extending saparticle and then we decide to remove those extensions and stgain datahub. Or when we do modifications on custom extensions and we deploy it.

With create-drop option I dont have any problem, but with ignore it doesnt work.

Any other suggestion?

Thanks

[Slava]

Most likely you have an extension, which customizes saparticle extension. The extension loading order is different and your extension with override=trueloads first but the original extension loads second, you get this error. Declare dependency in the overriding extension to saparticle. See how dependencies declared here. Also check Overriding Attributes just in case.

[Slava]

io,

if you pasted an example of the exception you're getting on restart, I would be more certain about the problem. However, I suspect you can ignore those exceptions.

I restarted my local DataHub in 'ignore' mode. Here is an example of one the exceptions I see:

 2018-03-02 10:13:34,638 [DEBUG] [c.h.d.e.l.i.CanonicalModelLoader] ValidationException{failures=[ValidationFailure{EXISTING_ITEM, rawAttribute: "Canonical attribute transformation already exists: TestCanonicalCategory, parent, TestRawCategory"}]}

This way Data Hub "warns" me that my extension loads an attribute (or item type), which already exists in the database. Of course it exists! It's exactly same extension I have loaded previously. However, in situations when DataHub loads extension from different vendors and there are duplicate item type or attribute names in those extensions, this warning may be very important and will indicate that things may not work as expected. In my case, of course, there is no problem, I can ignore those messages because DataHub is fully functional. The only time you have to be concerned, if you see something like ValidationException{failures=[ValidationFailure{FATAL,...}]}

Last note. When you restart in create-drop mode, the database is dropped and re-created. That wipes out all data from the metadata tables and therefore when the extension is loaded no duplicates found and no warnings given.

Hope this helps. If not, then paste the exception(s) you're seeing on restart.

[Former Member]

we have check this solution (kernel.autoInitMode=update) and it doesnt work.

Any other suggestion?

[VinayKumarS]

Currently data hub has below options..

 2016-10-24 12:07:24,994 [DEBUG] [c.h.d.p.j.u.i.DefaultJpaSchemaGenerationStrategy] Version table does not exist.
 2016-10-24 12:07:24,994 [WARN] [c.h.d.p.j.u.i.DefaultAutoInitPropertyMapper] If migrating, please refer to the following mapping between kernel and datahub properties values:
 ******************************************************************************************************
 * kernel.autoInitMode  * datahub.autoInitMode  * Schema Dropped  * Schema Created  * Schema Updated  *
 * *******************  * ********************  * **************  * **************  * **************  *
 * create-drop          * create-drop           * Yes             * Yes             * No              *
 * update               * create                * No              * If needed       * No              *
 * ignore               * ignore                * No              * No              * No              *
 ******************************************************************************************************
 

May be your problem is solved if you go with

 kernel.autoInitMode=update