Troubleshooting a Trfc production issue

[nabheetscn]

It's been a long time since I have written a blog. Through this blog I am going to share my understanding of Trfc's gained while handling a production issue.

Last week a strange ticket came with so many Process orders (COR2) getting stuck in SM58 Trfc.

The first look at the log informed us with below mentioned info

1. The function module struck in queue is a custom one....Custom code culprit :smile:
2. The status is of the call is SYSFAIL
3. Message in the error text is "Process order ABCD locked".

Table ARFCSSTATE screen shot attached below

Misunderstanding 1

One of the biggest misconceptions we had was Trfc function calls are started as Background task in update mode even when the locks acquired by SAP are not released.

Correct Understanding 1 ℹ

The understanding was corrected when we found in update debugging that ARFC are called as a new task at the end when all locks are released(Cross verified from SM12 :smile: ).Trfc's and Qrfc's are executed at the last via starting new task when all the locks have been released by the transaction. We were like fine if the locks are released why we still have locks.

So now the time was to check what was happening in our custom function module. The custom function module was called from an exit on save as a background task.   While replicating the scenario in testing system we found out that we had only one entry for our custom function module in SM58 with status as Transaction Recorded which is correct as the it was suppose to be executed.

Everything looked fine. Transaction locks are released and only one function module running in background task so from where does this issue coming from. So we decided to think in other direction. Even if their is an issue a background job should try to post these error entries. On  checking the system we found out there was no job scheduled for RSARFCEX to restart such LUW's. But we found out that there were so many background jobs created for other TRFC errored out entries in SM37. So we decided to look into this flip of the coin first to increase our understanding. :smile:

Misunderstanding 2

Why ARFC background jobs are not picking our entry but picking other entries to reprocess.

Correct Understanding 2 ℹ

Then the next thing which we stumble upon was different ARFC jobs getting created for program RSARFCSE in SM37 to process SM58 entries.  Why was this not happening for our case? On searching a bit about it we found one nicely documented SAP KBA 1902003 - Many ARFC* jobs in SM37 and many "Error when opening an RFC connection" in SM58 at the same time . 

We came to learn that only entries for which connection issues are encountered, the system creates the job automatically via settings maintained for the RFC destination in SM59.

Quick Fix( Temporary solution with root cause still unknown)

Stuck with no solution in sight except executing those LUW's or all the via report RSARFCEX or do some more custom development like storing entries in custom table and later process them via custom batch job :cry: :cry:

Good Luck Prevails :smile: Finally found the root cause

Till now we were stuck but have some temporary solution (not good type) in mind. The only thought was what can be locking the process order. On brain storming with one my colleague it came down to there can be two possible causes as mentioned below.

1. Either it is the user who are locking the process order by going back in the transaction and our Trfc calls are getting processed strongly. This we knew that there are very rare chances of this to happen.
2. Some other process running in parallel is blocking the stuff...May be some custom development which is not triggering in our development system.

So we decided to drill down more into the second option. We thought to check if there is any custom tables getting updated at the same time(this did the trick :smile: ). We stumble on one of SAP table DBTABLOG our savior which maintains the custom table log if table logging is active. We found some entries in one of the custom table were getting created. Checked in our culprit FM is it getting updated via us no..no..no..no... Bingo finally something else is also doing its part in the background which we are not able to replicate in testing system.

DBTABLOG

On doing a where used list of the table found another custom FM getting triggered in the same exit via back ground task same as ours.  On taking a detailed look at another FM, we found out that it was acquiring locks on the process order along with lot of wait statements.Since we knew from our initial learning that Trfc's are called after all locks have been released and they are called in parallel using starting new task. The second Z FM was not getting triggered as some conditions were not satisfied before the call.We simply triggered this second FM also in Testing client and we were able to replicate it.

Proposed Solution

1. Since both the Function module needs to be called lets call in them in Wrapper one with one call after another. No locking issue.
2. Convert this Trfc's call into a Qrfc call using queue which has it's pros and cons.

The solution is yet to be finalized. We will share them once we have started with it. Meanwhile you can also provide us your valuable ideas which you have implemented in similar kind of problems.

Takeaway from this issue

1. Trfc's and Qrfc's are called after the locks have been released. So this means we can use them when to perform further operations on the same object like PO/PR etc.
2. Why background jobs were not created for some failed Trfc's.
3. All Function module called in background task will remain in status Transaction recorded if you are in Update debugging mode and checking in SM58 with exception of the one being called in Update task( they will eventually get this status:)).

Please add your valuable opinion/comments for the same. Lets share lets learn :smile:

Common Syntax of Trfc call is

CALL FUNCTION IN BACKGROUND TASK DESTINATION