Datahub 6.5.0.7 not publishing orders

[Former Member]

Hello experts, we have an issue on prod system.

After a downtime for about a day (we stop datahub but not hybris ecommerce), we restarted datahub and have many PENDING publications for Orders (RawHybrisOrders to SALES_ORDERS IDOC)

For instance: GET /v1/pools/SAPORDER_OUTBOUND_POOL/publications/

we have

 <publicationActionData>
     <actionId>350502</actionId>
     <type>RETRY_ITEMS_PUBLICATION</type>
     <status>PENDING</status>
     <poolName>SAPORDER_OUTBOUND_POOL</poolName>
     <targetSystemPublications>
         <publicationId>2071905</publicationId>
         <status>PENDING</status>
         <targetSystemName>SapErpSystem</targetSystemName>
         <actionId>350502</actionId>
         <poolName>SAPORDER_OUTBOUND_POOL</poolName>
         <canonicalItemCount>0</canonicalItemCount>
         <internalErrorCount>0</internalErrorCount>
         <externalErrorCount>0</externalErrorCount>
         <ignoredCount>0</ignoredCount>
     </targetSystemPublications>
 </publicationActionData>
 <publicationActionData>
     <actionId>350500</actionId>
     <type>PUBLICATION</type>
     <status>PENDING</status>
     <poolName>SAPORDER_OUTBOUND_POOL</poolName>
     <targetSystemPublications>
         <publicationId>2071896</publicationId>
         <status>PENDING</status>
         <targetSystemName>SapErpSystem</targetSystemName>
         <actionId>350500</actionId>
         <poolName>SAPORDER_OUTBOUND_POOL</poolName>
         <canonicalItemCount>0</canonicalItemCount>
         <internalErrorCount>0</internalErrorCount>
         <externalErrorCount>0</externalErrorCount>
         <ignoredCount>0</ignoredCount>
     </targetSystemPublications>
 </publicationActionData>
 <publicationActionData>
     <actionId>350499</actionId>
     <type>PUBLICATION</type>
     <status>PENDING</status>
     <poolName>SAPORDER_OUTBOUND_POOL</poolName>
     <targetSystemPublications>
         <publicationId>2071895</publicationId>
         <status>PENDING</status>
         <targetSystemName>SapErpSystem</targetSystemName>
         <actionId>350499</actionId>
         <poolName>SAPORDER_OUTBOUND_POOL</poolName>
         <canonicalItemCount>0</canonicalItemCount>
         <internalErrorCount>0</internalErrorCount>
         <externalErrorCount>0</externalErrorCount>
         <ignoredCount>0</ignoredCount>
     </targetSystemPublications>
 </publicationActionData>

we have tried to publish with POST /v1/pools/SAPORDER_OUTBOUND_POOL/publications/

with payload:

{ "poolName": "SAPORDER_OUTBOUND_POOL", "type":"PUBLICATION", "targetSystemPublications": [ { "targetSystemName": "SapErpSystem" } ] }

and

{ "poolName": "SAPORDER_OUTBOUND_POOL", "type":"RETRY_ITEMS_PUBLICATION", "targetSystemPublications": [ { "targetSystemName": "SapErpSystem" } ] }

but I only get this trace

WARN c.h.d.s.p.i.DeletePublicationRetryEventListener - deleting 0 publication retries from target system id 1

So all orders from hybris are stopped in datahub. We are not able to delete/reprocess this publications. And what is most strange is that they have

0 0 0 0

Any help on how to delete/remove/avoid/resend this publications.

We have more than 3000 orders stopped in prod system.

In local.properties i have:

sapcoreconfiguration.pool=SAPCONFIGURATION_POOL sapcoreconfiguration.autocompose.pools=GLOBAL,SAPCONFIGURATION_POOL,SAPCONSUMER_INBOUND_POOL,SAPCONSUMER_OUTBOUND_POOL,SAPCONTACT_OUTBOUND_POOL,SAPCUSTOMER_INBOUND_POOL,SAPORDER_OUTBOUND_POOL,SAPORDER_INBOUND_POOL,SAPOAARSI_INBOUND_POOL sapcoreconfiguration.autopublish.targetsystemsbypools=GLOBAL.HybrisCore,SAPCONSUMER_OUTBOUND_POOL.SapErpSystem,SAPORDER_INBOUND_POOL.HybrisCore,SAPORDER_OUTBOUND_POOL.SapErpSystem,SAPCONSUMER_INBOUND_POOL.HybrisCore,SAPOAASITE_INBOUND_POOL.HybrisCore,SAPOAARSI_INBOUND_POOL.HybrisCore

datahub.publication.saveImpex=true

datahub.item.level.lifecycle.logging=true

datahub.max.publication.retry.count=1

Any suggestion will be appreciated.

Thans for your time.

io

[Former Member]

Many thanks Slava.

You were rigth. There was an IN_PROGRESS publication. After finding it and "delete" with PUT call and relaunching PENDING publications, finally the orders went to SAP.

Just a final question, I had lot of PENDING publications. Manually I tried to publish them, but they have as atribute equals to 0, so like if they hadn`t do a correct transformation, but as there were so many I decided to stop because it seems to have no end. But, some time after 1 or 2 hours all all orders went to SAP.

It has som kind of relationship with a retry or delete job in datahub. I have seen that there are some properties in local.host related with deleting publications. Can this be related?

Again, many many thanks.

[Slava]

I think the culprit of your problem is the publication stuck IN_PROGRESS. While there is an IN_PROGRESS publication no other publications can run in that pool. Even manual publications are simply queued with PENDING status. So, the more data you sent to DataHub the more PENDING publications accumulated. Once you killed the IN_PROGRESS publication, DataHub started picking up and processing those PENDING publications one after another and soon cleared the whole queue.

[rahulverma94]

I had a similar problem on GLOBAL pool. However after killing the culprit publication id, publications resumed. However strangely they again stopped without picking all the queued items. On restarting Datahub on create mode, the publications again resumed but again stopped after sometime. This kept happening...

As a last resort, we had to restart on create-drop and resend all the idocs. Any suggestion on the above behaviour of Datahub??

[Slava]

I have a feeling it's something in the environment. For some reason every so often datahub-adapter fails communicating the publication result back to datahub and that causes the problem.

It could be unstable network connection or, if containers are used, containers for datahub and the platform are not restarted together, some IP caching, etc. Hard to guess without carefully studying the problems. Check logs on the datahub-adapter side. Perhaps they can give you a clue.

[rahulverma94]

Thanks for responding! There are few observations: 1) There were 2 publications id which kept showing IN_PROGRESS(publication api response) after killing them.

2) After every restart on create, publications in api response never showed any newer IN_PROGRESS or PENDING status.

3) Still on every restart on create mode, queued publications kept happening only till sometime after which they stopped.

4) Strangely,callbacks from datahubadapter to datahub happened successfully with no errors on either side.

[Slava]

That is indeed strange.

On restart DataHub will try to recover publications. If it sees a publication IN_PROGRESS on restart, it will mark that publication as FAILED and will kick off the publication again.

[Former Member]

Thnsk Slave, I appreciate you answer. But I am still stuck

please could you provide me the exact query because I have tried bith with no results

GET /pools/SAPORDER_OUTBOUND_POOL/target-system-publications?q=status+eq+IN_PROGRESS or ..?q=status%3AIN_PROGRESS

also for PUT could you provide me the url ?

many thanks

[Slava]

Check out this answer https://answers.sap.com/questions/12761702/datahub-target-system-publication-items-gets-stuck.html?c...

It may help you

For statuses it should be: GET http://:/datahub-webapp/v1/target-system-publications?q=status:IN_PROGRESS

If it does not work try: GET http://:/datahub-webapp/v1/target-system-publications?q=status%3AIN_PROGRESS

[Slava]

As I explained in the comment to your answer it's very likely there is an IN_PROGRESS publication. Every target system needs to call back data hub to let it know the status of the publication. If that call does not happen, the publication stays IN_PROGRESS and blocks all other PENDING publications from being processed.

The callback does not happen possibly because of two reasons: network problems or the callback URL is misconfigured. You can easily check that:

GET /pools/SAPORDER_OUTBOUND_POOL/target-system-publications?q=status+eq+IN_PROGRESS or ..?q=status%3AIN_PROGRESS - don't remember exactly. This way you'll find the target system publication that is stuck and the response should have reference to publication ID, that is still IN_PROGRESS.

Similarly you can check for SUCCESS target system publications. If you find them - that means configuration is correct or at least was correct at some point. If you see no SUCCESS publications, most likely the configuration is incorrect.

OOTB DataHub uses datahub.server.url property for those callbacks, e.g datahub.server.url=http://127.0.0.1:9797/datahub-webapp/v1. The SAP extensions may use their own property. Search the documentation or ask team Griffin - they own those extensions.

Now, fixing the problem. Make sure the callback URL is correctly configured. Possibly restart DataHub. Now force complete the IN_PROGRESS publication: PUT /core-publications/{publicaitonId}, where publicationId is the target-system-publication ID that is IN_PROGRESS. Body of the request can be: {crashReport: "manually terminated", exportErrorDatas=[]}. This should put the publication to FAILED status and enable processing of other PENDING publications. The FAILED publication will be automatically re-attempted, since, looks like, you have retry publications enabled.

[Former Member]

Yes slava

I have found a publication in progress

     <actionId>349059</actionId>
     <type>PUBLICATION</type>
     <startTime>2019-05-20T19:09:12Z</startTime>
     <status>IN_PROGRESS</status>
     <poolName>SAPORDER_OUTBOUND_POOL</poolName>
     <targetSystemPublications>
         <publicationId>2055679</publicationId>
         <startTime>2019-05-20T19:09:12Z</startTime>
         <status>IN_PROGRESS</status>
         <targetSystemName>SapErpSystem</targetSystemName>
         <actionId>349059</actionId>
         <poolName>SAPORDER_OUTBOUND_POOL</poolName>
         <canonicalItemCount>0</canonicalItemCount>
         <internalErrorCount>0</internalErrorCount>
         <externalErrorCount>0</externalErrorCount>
         <ignoredCount>0</ignoredCount>
     </targetSystemPublications>
 </publicationActionData>

Is there a way to stop or delete this publication in order the rest can publish?

Regards.

[Slava]

The publication started today. I'm not sure how old it is in terms of running time, but is it really a stuck publication? Could it be DataHub is publishing but it's publishing much slower than the data arrive? Hence, is the PENDING publication accumulation?

[Former Member]

thanks slava

1. just in SAPORDER_OUTBOUND_POOL. Rest of pools work fine

2 with GET v1/pools/SAPORDER_OUTBOUND_POOL/publications/ I only get 50 items, but no one is IN_PROGRESS

1. their status still pending. I am launching POST call /v1/pools/SAPORDER_OUTBOUND_POOL/publications/ with payload { "poolName": "SAPORDER_OUTBOUND_POOL", "type":"PUBLICATION", "targetSystemPublications": [ { "targetSystemName": "SapErpSystem" } ] }

and

{ "poolName": "SAPORDER_OUTBOUND_POOL", "type":"RETRY_ITEMS_PUBLICATION", "targetSystemPublications": [ { "targetSystemName": "SapErpSystem" } ] }

but I only get this trace

WARN c.h.d.s.p.i.DeletePublicationRetryEventListener - deleting 0 publication retries from target system id 1

If you can give me more clues I will appreciate.

thanks

[Slava]

The more I think about this issue the more I'm getting a feeling that there must be IN_PROGRESS publication. The problem it's not easy to find it, if you have many PENDING publications, because unfortunately there is no a REST call that would return publications by their statuses.

When you called GET v1/pools/SAPORDER_OUTBOUND_POOL/publications/, you received 50 publications but that's only first page of the response. it's identical to sending this request: GET v1/pools/SAPORDER_OUTBOUND_POOL/publications?pageNumber=0&pageSize=50. You can query for more publications by changing those parameters.

Do you have access to the database? It might be just easier to query against the PublicationAction table.

A little bit of context. First, I thought it might be a misconfiguration among the other reasons, but in that case publication actions would not be even created. So, that most likely reason for PENDING actions being accumulated is presence of an IN_PROGRESS publication, which blocks processing of those publication actions in that pool.

[Slava]

io, I'm not an expert in those sap extensions configuration, but we can try to troubleshoot your problem. But before recommending a solution I need to understand what's possible causing the problem, so I have a few questions:

1. Are PENDING publications in a specific data pool or in many different data pools?

2. Do you see a publication with IN_PROGRESS status for that pool(s)? If so, does it seem running unreasonable long?

3. What happened with those publications that you attempted to trigger manually by sending POST request? What's their status?