java.lang.OutOfMemoryError: Java heap space during 'ant updatesystem -DdryRun=true'

[Former Member]

Hello,

I am upgrading our Hybris platform from 5.1.0 to 5.6.0. Our current UAT is running on Oracle and has been initialized about 1.5 years ago. We've been working on 5.1.0 ever since.

Now I am trying to upgrade its type system to Hybris 5.6, but I am getting out of memory exceptions each time. Here's the last error from 'ant updatesystem -DdryRun=true':

 [yTypeSystemUpdate] 11:14:21,333 [main] INFO  DbTypeSystemImpl - Fetching type system related entities
 [yTypeSystemUpdate] 11:14:22,877 [main] INFO  DbTypeSystemImpl - 454 deployments have been fetched in 1.541 s
 [yTypeSystemUpdate] 11:14:23,489 [main] INFO  DbTypeSystemImpl - 1356 composedtypes have been fetched in 611.6 ms
 [yTypeSystemUpdate] 11:14:31,028 [main] INFO  DbTypeSystemImpl - 31361 attributess have been fetched in 7.539 s
 [yTypeSystemUpdate] 11:14:31,049 [main] INFO  DbTypeSystemImpl - 22 atomictypes have been fetched in 20.18 ms
 [yTypeSystemUpdate] 11:14:31,227 [main] INFO  DbTypeSystemImpl - 582 collectiontypes have been fetched in 177.7 ms
 [yTypeSystemUpdate] 11:14:31,244 [main] INFO  DbTypeSystemImpl - 42 maptypes have been fetched in 16.47 ms
 [yTypeSystemUpdate] 11:14:31,557 [main] INFO  DbTypeSystemImpl - 1932 enumerationvalues have been fetched in 313.2 ms
 [yTypeSystemUpdate] 11:14:31,605 [main] INFO  DbTypeSystemImpl - 333 numberseries have been fetched in 47.53 ms
 
 BUILD FAILED
 /app01/hybris/bin/platform/build.xml:169: The following error occurred while executing this line:
 /app01/hybris/bin/platform/resources/ant/platformadministration.xml:171: java.lang.OutOfMemoryError: Java heap space
     at java.util.Arrays.copyOf(Arrays.java:3332)
     at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
     at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
     at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421)
     at java.lang.StringBuilder.append(StringBuilder.java:136)
     at de.hybris.bootstrap.ddl.dbtypesystem.HashGenerationStrategy$StrategyTemplate.getHashFor(HashGenerationStrategy.java:411)
     at de.hybris.bootstrap.ddl.dbtypesystem.impl.DbProps.<init>(DbProps.java:36)
     at de.hybris.bootstrap.ddl.dbtypesystem.impl.DbTypeSystemImpl.fetchProps(DbTypeSystemImpl.java:313)
     at de.hybris.bootstrap.ddl.dbtypesystem.impl.DbTypeSystemImpl.fetchEntities(DbTypeSystemImpl.java:207)
     at de.hybris.bootstrap.ddl.dbtypesystem.impl.DbTypeSystemImpl.initialize(DbTypeSystemImpl.java:158)
     at de.hybris.bootstrap.ddl.dbtypesystem.impl.DbTypeSystemFactory.createDbTypeSystem(DbTypeSystemFactory.java:35)
     at de.hybris.bootstrap.ddl.HybrisSchemaGenerator.createDbTypeSystem(HybrisSchemaGenerator.java:303)
     at de.hybris.bootstrap.ddl.HybrisSchemaGenerator.update(HybrisSchemaGenerator.java:211)
     at de.hybris.ant.taskdefs.init.TypeSystemUpdaterTask.executeSchemaGenerator(TypeSystemUpdaterTask.java:30)
     at de.hybris.ant.taskdefs.init.AbstractInitUpdateTask.execute(AbstractInitUpdateTask.java:50)
     at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:292)
     at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
     at java.lang.reflect.Method.invoke(Method.java:497)
     at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
     at org.apache.tools.ant.Task.perform(Task.java:348)
     at org.apache.tools.ant.taskdefs.Sequential.execute(Sequential.java:68)
     at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:292)
     at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
     at java.lang.reflect.Method.invoke(Method.java:497)
     at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
     at org.apache.tools.ant.Task.perform(Task.java:348)
     at org.apache.tools.ant.taskdefs.MacroInstance.execute(MacroInstance.java:396)
     at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:292)
     at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 
 Total time: 41 minutes 20 seconds

The out of memory errors always happen around fetching DB data by DbTypeSystemImpl. My current instance is configured with

 tomcat.generaloptions=-Xms9G -Xmx9G -Xss256K -XX:+UseG1GC -XX:+DisableExplicitGC -XX:+OptimizeStringConcat -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="${bundled.server.access.log}" -Xloggc:"${bundled.server.access.log}"/java_gc.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dorg.tanukisoftware.wrapper.WrapperManager.mbean=true -Djava.endorsed.dirs="%CATALINA_HOME%/lib/endorsed" -Dcatalina.base=%CATALINA_BASE% -Dcatalina.home=%CATALINA_HOME% -Dfile.encoding=UTF-8 -Dlog4j.configuration=log4j_init_tomcat.properties -Djava.util.logging.config.file=jdk_logging.properties -Djava.io.tmpdir="${HYBRIS_TEMP_DIR}" -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv6Addresses=false

I've also tried to run update on a system with 12GB of memory and got a bit further:

 [de.hybris.bootstrap.ddl.dbtypesystem.impl.DbTypeSystemImpl] 7413627 props have been fetched in 20.36 min

 tomcat.generaloptions=-Xms12G -Xmx12G -XX:PermSize=500M -Xss256K -XX:+UseG1GC -XX:+DisableExplicitGC -XX:+OptimizeStringConcat -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="${bundled.server.access.log}" -Xloggc:"${bundled.server.access.log}"/java_gc.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dorg.tanukisoftware.wrapper.WrapperManager.mbean=true -Djava.endorsed.dirs="%CATALINA_HOME%/lib/endorsed" -Dcatalina.base=%CATALINA_BASE% -Dcatalina.home=%CATALINA_HOME% -Dfile.encoding=UTF-8 -Dlog4j.configuration=log4j_init_tomcat.properties -Djava.util.logging.config.file=jdk_logging.properties -Djava.io.tmpdir="${HYBRIS_TEMP_DIR}" -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv6Addresses=false

The problem seems to be that our DB type system is too large. Is there a way to clean it up before the upgrade?

Thanks!

[Former Member · Accepted solution]

Check the size props table, it's a famous-ish offender here. That specific issue has been resolved since 5.1.

When we contacted hybris about our props table issue (if that is your issue: it seems like your second log with 12GB heap gets OOM error while fetching props table), they suggested doing the following in this order:

1. Clean up SavedValues table

2. Clean up SavedValuesEntries table

3. Truncate props table

In our case, this had adverse affects, as while the update succeeded, we lost a lot of WCMS data in the process. I suggest backing up your db if you're gonna try this.

[Former Member · Accepted solution]

Thanks for your suggestion, Michael. We've got through the problem by bumping up the heap to 18GB before you replied.

[Former Member · Accepted solution]

I did a bit of research in suggested direction and we have a cleanup story based on

• https://jira.hybris.com/browse/PLA-6398

• https://jira.hybris.com/browse/PLA-11978

• https://jira.hybris.com/browse/PLA-6701

• https://wiki.hybris.com/display/release5/SavedValues+-+Keeping+Track+of+Attribute+Value+Modification...

• https://wiki.hybris.com/display/release5/cronjob+-+Technical+Guide?focusedCommentId=203490000#cronjo...