Backoffice context occasionally fails to startup: NullPointerException at DomLoader.java:134

[pavelska]

Hybris 18.08.3 is running on a cluster. One of the nodes is dedicated to background jobs and is not exposed publicly for visitors. Most cron jobs are running on that node. Lately, often (not always) after restarting this background node, following exception gets thrown: NPE stack trace

After that some classes become inaccessible - lots of NoClassDefFoundError, for example: NoClassDefFoundError example

Rebuild/restart helps, but only in roughly 70% of cases. Sometimes we have to restart the server several times before NoClassDefFoundError disappear.

What specifically could cause it and how could it be mitigated?

Any help is very much appreciated.

UPDATE 2020-06-03: Workaround found

We have discovered that if we disable the frequently running backoffice index cron job (which runs every minute) during server startup, then the mentioned error does not happen. After server fully starts up, then we can re-enable the job, and it runs normally.

Our current guess is that because of high frequency of this job, it tries to run too early, while the server has not completely started up. So the job might try to use some classes before those classes are loaded. And if the first failed attempt to load those classes fails, then future attempts also fail (still not sure what exact mechanism is responsible for that).

12526-classloaderutils-stacktrace.txt

12525-bo-context-npe-stacktrace.txt

[pavan_joshi1]

From stack trace, I could only make out that :
some line in cockpit configuration (may be backoffice *-config.xml) is wrong.
Its parsing is expecting something and hence causing NPE since missing.

Please validate your config XMLs from all related extensions.

Might help!
Thanks!

[pavelska]

Thanks for looking into this question. The backoffice config XML is quite heavily customized in my project, and having very little information in the error message, I'm afraid it will be very difficult to find the problem (assuming it's there in the first place). But I will give it a look.

Current working hypothesis is that issue might be related to this: https://answers.sap.com/questions/12761421/solr-search-in-backoffice-null-pointer-exception-d.html. Next thing I am going to try out when I get back to this problem is the workaround described in mentioned Q&A thread.

[pavelska]

An elegant workaround was discovered:

task.polling.startup.delay.enabled=true

From SAP docs:

Postponing Database Polling

Platform may start to poll the database to retrieve tasks during Tomcat startup even when not all servlet contexts are initialized. To postpone polling the database for tasks until all servlet contexts are started, set the task.polling.startup.delay.enabled property to true. The default value of this property is false.

Source: https://help.sap.com/viewer/d0224eca81e249cb821f2cdf45a82ace/1808/en-US/7946d61720754981887ba63af9d8...

[pavelska]

Update: turns out that this solution does not solve the problem - it still keeps happening from time to time.