Datahub Extensions not getting loaded in Datahub startup logs in Tomcat

[Former Member]

Hi Experts

I have followed Hello World trail to setup datahub but i am not able to see my hello-raw-datahub-extension.jar , canonical or target jar files in logs when data base is getting loaded. Can anyone suggest why is this happening?

My Local properties file has the following info:

datahub.cleanup.rawitem.enabled=true datahub.cleanup.canonicalitem.enabled=true datahub.cleanup.targetitem.enabled=true datahub.cleanup.publisheditems.enabled=true

datahub.security.basic.admin.user=admin datahub.security.basic.admin.password=nimda datahub.security.basic.read_only.user=admin1 datahub.security.basic.read_only.password=nimda

datahub.core.export.service.clientid=datahubTestID datahub.core.export.service.clientsecret=secret

datahub.encryption.key.path=C:/opt/datahub/config/encryption.txt

datahubadapter.datahuboutbound.user=admin datahubadapter.datahuboutbound.password=nimda

datahub.secure.data.masking.mode=true

datahub.secure.data.masking.value=*** datahub.security.https.enabled=true datahub.server.url=https://localhost:8443/datahub-webapp/v1

dataSource.className=com.mysql.jdbc.jdbc2.optional.MysqlDataSource dataSource.jdbcUrl=jdbc:mysql://localhost:3306/integration?useConfigs=maxPerformance&rewriteBatchedStatements=true&characterEncoding=utf8 dataSource.username=hybris dataSource.password=hybris

dataSource.jndiName=integration

And Below is my Datahub webapp xml file placed in Tomcat

   <PostResources className="org.apache.catalina.webresources.DirResourceSet"
                 base="C:/opt/datahub/config" internalPath="/" webAppMount="/WEB-INF/classes" />
   <PostResources className="org.apache.catalina.webresources.DirResourceSet"
                 base="C:/opt/datahub/extensions/" webAppMount="/WEB-INF/lib" />
 </Resources>
 

[Former Member · Accepted solution]

Hey Justin

Thanks for all the support and guiding me through this. I figured out my mistake, it was in the jar that i was creating, documenting here my silly mistake so that someone else does not waste time on it..:) When i created Jar using hello-raw-datahub-extension.xml, we place it in our own created META-INF folder and run "jar cf hello-raw-datahub-extension.jar META-INF" . So basically we need to run this command in the directory outside the Meta-INF folder so that that xml gets included in the jar file. My mistake was i was doing that Inside the META-INF folder and thus jar was getting created but it contained nothing. Once I created it from outside the folder proper jar with xml got created and logs reflected the new extensions, even when they were placed in /opt/datahub/extensions

Thanks and Regards Rahul

[former_member333910]

You don't mention which version of Tomcat you are using. The datahub-webapp.xml you pasted above should work for Tomcat 8.5.x. If you are using Tomcat 7.x this format is different. The v6 docs show the format needed for Tomcat 7.x here: Making Extensions Accessible

Data Hub uses standard Tomcat functionality for loading jars from an outside directory. Therefore the Tomcat docs should provide additional info in case you still have problems.

As an alternative, you can deploy your custom extension jars directly in the /datahub-webapp/WEB-INF/lib directory of the deployed Data Hub. This can be done using the Overlay instructions or manually if needed.

[Former Member]

Hi Justin

Thanks for your response. You are correct, I am using Tomcat 8.5.32 I tried placing the extension jars in /datahub-webapp/WEB-INF/lib, still couldn't see them in logs getting picked up. Although with the server running I removed it from above location and placed it in another location, on doing so I got windows prompt that it is being used by Java application, which i think is an indication that it might be getting picked on server start just not showing in logs. Please let me know if my understanding if correct?

[Former Member]

But even if gets loaded when i try to load data from postman using below URL https://localhost:8443/datahub-webapp/v1/data-feeds/DEFAULT_FEED/items/RawHelloWorldProduct

It gives the following response: [ { "message": "RawHelloWorldProduct type is not a valid RawItem type", "propertyName": "itemType", "failureType": "FATAL" } ]

[former_member333910]

Rahul, something doesn't sound quite right. You can try posting the Data Hub tomcat log. Perhaps I can provide more feedback after I've reviewed it. If your jars are found in the classpath then Data Hub should certainly be logging something.

You posted an earlier issue about Tomcat 8.5.32 which I recollect we were able to resolve with a workaround. However, considering your current issue you may want to try with an earlier Tomcat version just to confirm the workaround is indeed functioning as expected.

[Former Member]

Hey Justin, Yes I asked the question that we were able to resolve using a workaround.BUt then Tomcat was not at all picking up anything of datahub, but now when I hit https://localhost:8443/datahub-webapp/v1/version I get the datahub version that is running with Tomcat server, plus logs show that datahub is loaded by the server and in MySQL as well i see the tables created after server startup in my schema.

Still extensions not getting picked. Please find attached logs on server startup.link text

[Former Member]

when I check with following URL i get response as expected: https://localhost:8443/datahub-webapp/v1/data-feeds/DEFAULT_FEED

Response:

 <feedId>0</feedId>
 <name>DEFAULT_FEED</name>
 <description>Default feed</description>
 <poolingStrategy>GLOBAL_POOL</poolingStrategy>

[former_member333910]

Rahul, thanks for uploading the startup log. I've reviewed it and I see no evidence that Data Hub is finding any of your custom extensions during startup.

For example the log shows Data Hub finding the following extensions which are shipped with Data Hub: csv-web-service, datahub-cleanup, etc.

For each of those extensions Data Hub logs that it is loading from C:/apache-tomcat-8.5.32/webapps/datahub-webapp/WEB-INF/lib/

Are you certain your extensions are in that directory?