Why doesn't the provided solr indexer job include a default CronJobProgressTracker?

[Former Member]

Hello,

I am wondering why isn't a default de.hybris.platform.cronjob.jalo.CronJobProgressTracker provided for the default de.hybris.platform.solrfacetsearch.indexer.cron.SolrIndexerJob ? I mean that is a really useful feature to be able to follow the progression of the index update or creation to see how many items have been processed and when it is estimated to be done. It isn't very easy to actually add it by us on top of the original implementation, especially going in to the index service but if that is what it takes..

Sure we will do it but wouldn't it be better if Hybris provided a solution for tracking by default.

Regards

[Former Member · Accepted solution]

Hello community.

Let's do some coding. You'd be surprised how amazing it is to get the solr index progression on both the cron job history models and in the logs.

What I have done to avoid having to pass the tracker around is that I created a SolrTracker singleton bean. I have also added a precondition that if a SolrIndexerJob is already running then we don't run another until that one is finished or aborted. This precondition makes it so that I don't have to lock the tracker from potential concurrent R/W.

I will call my tracker bean SolrProgressTracker. What the details of the code looks like is not that important. Instead we will just go over the basic methods with some Java DOC.

     /**
      * Create new CronJobProgressTracker if it doesn't already exist
      */
     public void createNewTracker(final CronJobModel cronJobModel)
 
     /**
      * Initialize tracker bean state with:
      * 
      * - total items to create documents for
      * - the incremental percentage for each item
      */
     public void initializeTracker(final int totalCount)
 
     /**
      * Increment total documents created and append one more progress increment
      *
      * Log start, progress, processed, total, processingPerItem, estimatedFinish
      */
     public void addProgress()
 
     /**
      * Close the CronJobProgressTracker and log final statistics
      */
     public void finish()

Then we just bind this bean:

 <bean id="solrProgressTracker" class="....SolrProgressTracker" scope="singleton" />

Now we have to make use of this cool new tracker bean. What we need to do is override these default beans:

1. SolrIndexerJob

2. DefaultIndexerStrategy

3. ItemIdentityProvider

Beans:

 <bean id="solrIndexerJob" class="...TrackedSolrIndexerJob" parent="abstractIndexerJob" >
             <property name="flexibleSearchService" ref="flexibleSearchService"/>
             <property name="modelService" ref="modelService"/>
             <property name="sessionService" ref="sessionService"/>
             <property name="indexerService" ref="indexerService" />
             <property name="indexer" ref="indexer" />
         </bean>
 <bean id="itemIdentityProvider" class="...TrackingItemIdentityProvider">
         <property name="modelService" ref="modelService" />
         <property name="catalogTypeService" ref="catalogTypeService" />
     </bean>
 
         <alias name="trackingIndexerStrategy" alias="indexerStrategy" />
         <bean id="trackingIndexerStrategy" class="...TrackingIndexerStrategy" parent="defaultIndexerStrategy"/>

Extended SolrIndexerJob

 public class TrackedSolrIndexerJob extends SolrIndexerJob {
     
         @Resource(name = "solrProgressTracker")
         private SolrProgressTracker solrProgressTracker;
     
         @Override
         public PerformResult performIndexingJob(CronJobModel cronJob) {
             solrProgressTracker.createNewTracker(cronJob);
             try {
                 return super.performIndexingJob(cronJob);
             } finally {
                 solrProgressTracker.finish();
             }
         }
     }

 

Then the IndexerStrategy

 public class TrackingIndexerStrategy extends DefaultIndexerStrategy {
 
     @Resource(name = "solrProgressTracker")
     private SolrProgressTracker solrProgressTracker;
 
     @Override
     protected void doExecute(IndexerContext indexerContext) throws IndexerException {
         solrProgressTracker.initializeTracker(indexerContext.getPks().size());
         super.doExecute(indexerContext);
         solrProgressTracker.setProgress(100.0);
     }
 }

Finally the ItemIdentityProvider

 public class TrackingItemIdentityProvider extends ItemIdentityProvider {
 
      @Override
      public String getIdentifier(final IndexConfig indexConfig, final ItemModel itemModel) {
          super.getIdentifier(indexConfig, itemModel);
          try {
              solrProgressTracker.addProgress();
          } catch (final Exception ignored) {
              // Log
          }
      }

Now you can follow in real time the solr index job building documents and also be able to see the progress and calculated an estimated time it will finish. Here is an example log from my tracker bean while it is running:

2019-01-16T12:25:36,165 INFO SolrProgressTracker - feature='Solr', desc='Solr index progress', start='2019-01-16T11:20:25', progress='78.5460%', processed=#46680, total=#59094, processingPerItem='0.0838s', estimatedFinish='2019-01-16T12:42:56.165'

Who needs Hybris to implement awesome features when you have the power yourself.

Have a nice day everyone.

[Former Member · Accepted solution]

Friendly reminder! If you are running with multi threading, like I do you have to in your tracker bean realize the concurrency situation when you execute solrProgressTracker.addProgress()

[former_member224482 · Accepted solution]

Looks great in the log.

Is there any UI in backoffice to see the progress of a running cron job ?

[nicolabeghin · Accepted solution]

super-interesting, thanks a lot!

[Former Member · Accepted solution]

Yes there is, there is a processes tab since I think Hybris 6.7. Here is the official documentation: https://help.hybris.com/1811/hcd/9a3308ccdc6e4ece9386dcceda4fb309.html

You can add processes that should be shown there by doing this:

 <util:map id="customBackofficeCronJobHistoryIncludes" key-type="java.lang.String"
           value-type="de.hybris.platform.servicelayer.cronjob.CronJobHistoryInclude">
     <entry key="syncProcesses">
         <bean class="de.hybris.platform.servicelayer.cronjob.CronJobHistoryInclude">
             <property name="jobTypeCode" value="#{T(de.hybris.platform.catalog.model.SyncItemJobModel)._TYPECODE}"/>
         </bean>
     </entry>
     <entry key="solrProcesses">
         <bean class="de.hybris.platform.servicelayer.cronjob.CronJobHistoryInclude">
             <property name="cronJobTypeCode"
                       value="#{T(de.hybris.platform.solrfacetsearch.model.indexer.cron.SolrIndexerCronJobModel)._TYPECODE}"/>
         </bean>
     </entry>
     <entry key="celumProcesses">
         <bean class="de.hybris.platform.servicelayer.cronjob.CronJobHistoryInclude">
             <property name="jobTypeCode" value="#{T(com.papyrus.bzr.mam.model.cronjobs.PyMediaIntegrationJobModel)._TYPECODE}"/>
         </bean>
     </entry>
 </util:map>
 <cng:property-extender bean="defaultProcessesFacade" property="cronJobHistoryIncludes">
     <ref bean="customBackofficeCronJobHistoryIncludes"/>
 </cng:property-extender>
 <cng:property-extender bean="defaultBackofficeAfterCronJobFinishedEventListener" property="processesIncludes">
     <ref bean="customBackofficeCronJobHistoryIncludes"/>
 </cng:property-extender>
 <cng:property-extender bean="defaultBackofficeBeforeCronJobStartEventListener" property="processesIncludes">
     <ref bean="customBackofficeCronJobHistoryIncludes"/>
 </cng:property-extender>
 

We are currently running Hybris 6.7 and I had to add my own renderer to get the progressbar displayed. An additional thing you can do is that instead of just using a standard CronJobProgressTracker in the solr tracker bean you can implement a special SolrCronJobProgressTracker for solr taking inspiration from CatalogVersionSyncProgressTracker so that it holds the total items to process and items that have been processed so far. Then you can in your custom renderer for the Solr process output these values under the progressbar of the solr job.

Very cool.

Regards

[Former Member · Accepted solution]

The defaultProcessesFacade bean is what extracts the cron job histories from certain jobs and cron jobs you specified in the BackofficeCronJobHistoryIncludes map.

The defaultBackofficeAfterCronJobFinishedEventListener is the listener responsible for highlighting when a running job completeda and defaultBackofficeBeforeCronJobStartEventListener will initialize a view in the processes sidebar rendering an item for the process that just started running.

Without those listeners the processes sidebar will not be immediately updated with new and finished process items. Then you would have to close and open it again making the initial request go through the defaultProcessesFacade again.

Worth noting though is that the more jobs you add the slower the initial render of the processes sidebar but once loaded the listeners will without delay make sure the list is up to date in real-time with the running processes which is very useful.