OTB cleanup extension not deleting raw and canonical records post successful publication

[nkjahan1388]

We are using Hybris 6.6. We use Datahub to import data from ERP system. We have the cleanup properties enabled and reduced delay to 180mins as mentioned below

• datahub.cleanup.rawitem.enabled=true

• datahub.cleanup.canonicalitem.enabled=true

• datahub.cleanup.targetitem.enabled=true

• datahub.cleanup.publisheditems.enabled=true
  

• datahub.cleanup.canonicalitem.time.delay=10800

• datahub.cleanup.publisheditems.time.delay=10800

However we noticed that raw and canonical items are piling up in database and not being deleted even after successful publication. We once even faced the pool size exceeded the max limit exception due to this. In the logs , I don't see any delete query fired for deleting raw/canonical items. To test this I reduced the datahub.cleanup.canonicalitem.time.delay and datahub.cleanup.publisheditems.time.delay to 108. Any thoughts or suggestions? Following are the logs :- Following are the logs 018-04-03 15:24:13,049 [INFO] [c.h.d.s.i.AsynchronousEventPublicationService] Firing data hub event : com.hybris.datahub.api.event.DeletePublicationRetryEvent@22009204[] 2018-04-03 15:24:13,074 [DEBUG] [c.h.d.p.i.DefaultProcessMonitor] Unregistered Publication #1, 0 processes running 2018-04-03 15:24:13,075 [INFO] [c.h.d.s.i.AsynchronousEventPublicationService] Firing data hub event : PublicationCompletedEvent{actionId=1, poolId=22000} 2018-04-03 15:24:13,127 [INFO] [c.h.d.e.l.TargetSystemPublicationCompletedEventListener] TargetSystemPublicationCompleted: SAPPRODUCT_INBOUND_POOL Publication ID: 1 Status: COMPLETE_W_ERRORS 2018-04-03 15:24:13,131 [WARN] [c.h.d.s.p.i.DeletePublicationRetryEventListener] deleting 0 publication retries from target system id 10 2018-04-03 15:24:13,141 [INFO] [c.h.d.s.p.i.DeletePublicationRetryEventListener] deleting publication retries for the following canonical items [] in pool 22000 for target system id 10 2018-04-03 15:24:13,156 [INFO] [c.h.d.p.i.DefaultAutoPublicationDecisionStrategy] Performing auto publish on pool SAPPRODUCT_INBOUND_POOL to target system(s) [HybrisAlconProduct]. 2018-04-03 15:24:13,187 [INFO] [c.h.d.p.e.AutoRetryPublicationEventListener] Pool 'SAPPRODUCT_INBOUND_POOL' has no Pending or InProgress Compositions, creating Retry Publication to process retry publication failures 2018-04-03 15:24:13,193 [INFO] [c.h.d.s.i.AsynchronousEventPublicationService] Firing data hub event : InitiatePublicationRetryEvent{poolId=22000, targetSystemNames=[HybrisAlconProduct]} 2018-04-03 15:24:13,267 [INFO] [c.h.d.s.i.AsynchronousEventPublicationService] Firing data hub event : PerformPublicationRetryEvent{actionId=2, poolId=22000, systemUrl='http://localhost:7070/datahub-webapp/v1', targetPublicationIds=[7]} 2018-04-03 15:24:13,277 [DEBUG] [c.h.d.p.i.DefaultProcessMonitor] Registered Publication #2: 1 processes running 2018-04-03 15:24:13,278 [INFO] [c.h.d.s.i.AsynchronousEventPublicationService] Firing data hub event : PublicationStartedEvent{actionId=2, poolId=22000} 2018-04-03 15:24:13,321 [DEBUG] [c.h.d.s.i.DefaultPublicationActionService] Pinging target system HybrisAlconProduct at http://localhost:9001/datahubadapter for target system publication id 7 and pool SAPPRODUCT_INBOUND_POOL. 2018-04-03 15:24:13,471 [DEBUG] [c.h.d.s.p.i.DefaultCanonicalMetadataDependencyResolver] Creating target items from canonical item types in order: [CanonicalItemMetadataEntity{id=83, itemMetadataId=84, itemType='AlconCanonicalProduct'}, CanonicalItemMetadataEntity{id=85, itemMetadataId=86, itemType='CanonicalBaseProductUnit'}, CanonicalItemMetadataEntity{id=89, itemMetadataId=90, itemType='CanonicalSalesUnit'}, CanonicalItemMetadataEntity{id=87, itemMetadataId=88, itemType='CanonicalMarketStatus'}] 2018-04-03 15:24:13,471 [INFO] [c.h.d.s.i.AbstractTargetItemService] Creating Target Items for type: 'AlconCanonicalProduct', and Target System: HybrisAlconProduct 2018-04-03 15:24:13,503 [INFO] [c.h.d.s.i.AbstractTargetItemService] Creating Target Items for type: 'CanonicalBaseProductUnit', and Target System: HybrisAlconProduct 2018-04-03 15:24:13,514 [INFO] [c.h.d.s.i.AbstractTargetItemService] Creating Target Items for type: 'CanonicalSalesUnit', and Target System: HybrisAlconProduct 2018-04-03 15:24:13,516 [INFO] [c.h.d.s.i.AbstractTargetItemService] Creating Target Items for type: 'CanonicalMarketStatus', and Target System: HybrisAlconProduct 2018-04-03 15:24:14,684 [INFO] [c.h.d.s.i.AbstractTargetItemService] Creating Target Items for type BaseMarketAvailabilityRules from 2 Canonical Items in Pool SAPPRODUCT_INBOUND_POOL 2018-04-03 15:24:14,716 [INFO] [c.h.d.s.i.AbstractTargetItemService] Creating Target Items for type MarketAvailabilityRules from 2 Canonical Items in Pool SAPPRODUCT_INBOUND_POOL

[nkjahan1388]

Any thoughts?

[rafal_gdowski]

What are the types of canonical items that are not getting cleaned up and their publication statuses?

[rafal_gdowski]

Can you verify that your properties are set properly by enabling property logging as described here:

https://help.hybris.com/6.6.0/hcd/599465be2dd14ce3956b50ea8a2b90ed.html

This should help you see whether your properties are loaded on startup.

[Former Member]

Hi All, Even I am facing the same issue in my project, could you please help ?

Regards, Gaurav

[rafal_gdowski]

Have you verified that your properties are set as described above?