Question Re: Dynamic Parallel Processing limitation in Technology Q&A

Dynamic Parallel Processing limitation

Former Member — Thu, 18 Jun 2015 14:37:00 GMT

Hello,

We have a workflow that starts whenever a material change. The workflow then finds all active PO's that are involved with this material and using the dynamic parallel processing feature, submits a subworkflow for each PO, to update it with the material changes.

The issue I'm having is that once the PO's reach 1000, workflow goes into error saying that it can only handle 999 parallel instances. Is this a "set in stone" feature? What can I do to process more than 999 instances?

Thanks!

Dan Stoicof

Re: Dynamic Parallel Processing limitation

Former Member — Thu, 18 Jun 2015 16:19:50 GMT

Don't use dynamic parallel processing for 1000 entries. Or if you really need to do it, please explain the reason behind it. I would never do that kind of parallel processing inside a single workflow instance. Couldn't you handle this by starting 1000 separate workflows? Does the workflow need ti handle all the 1000 entries/cases/subworkflows (whatever) before the actual main workflow can be completed? If yes, why?

I think there there is some kind of internal limit of 9999 nodes per workflow, but I have never seen if there is also a limit for 999 parallel instances. In any case I believe that you are right, and there is a limit. Anyhow, even if there wasn't such a limit of 999, I would still find an alternative solution to handle the functionality in some other way than using so many parallel instances.

Regards,

Karri

Re: Dynamic Parallel Processing limitation

SandySingh — Thu, 18 Jun 2015 23:05:24 GMT

Hello

You can consider a change in design :

Read all the PO's related to the Material and approvers for these PO's in a background step in the workflow. Download a PO list as PDF or Excel and attach to single approval workitem or email step.

Once approved or notified; you can call another background step to update the PO with material changes.

Regards

Sandy

Re: Dynamic Parallel Processing limitation

anjan_paul — Fri, 19 Jun 2015 07:54:57 GMT

Hi,

I am completely agree with Karri and Sandy. Change the design is the best option

Re: Dynamic Parallel Processing limitation

Former Member — Mon, 22 Jun 2015 17:21:15 GMT

Thank you all for your responses. I understand your position, to redesign the workflow.

Let me give you my take on this: This workflow has no approvers, it's just meant to automate updating of PO's (and PR's) when the material changes. I see no harm starting a dynamic parallel process for as many items as there are found. The reason is that the volume is managed by the Event Queue. This has been working just fine, with no issues what-so-ever. That was until the workflow stopped because it reached the "limit" of 999 items (probably there is a 3 digit field defined somewhere).

I welcome your comments/suggestions.

Regards,

Dan Stoicof

Re: Dynamic Parallel Processing limitation

Former Member — Mon, 22 Jun 2015 18:16:04 GMT

> I see no harm starting a dynamic parallel process for as many items as there are found.

Even after you found out that it doesn't work?

>The reason is that the volume is managed by the Event Queue.

How? Your workflow gets started by an event, right? That part is managed by the event queue. But once the workflow has started, I don't see how the event queue would be managing anything?

I would definitely not use parallel processing in your scenario. Normally when you want to use parallel processing, there should be a reason for that (something needs to be done in parallel - for example an invoice should be approved simultaneously by several agents etc.). Why should the POs get updated in parallel? Why it cannot be done sequentially (which actually happens already even when you are using the parallel processing).

I am not actually sure if I would use the workflow at all - or at least not create individual work items for each and every PO/PR. Seems like unnecessary overhead for me to use workflow to handle this kind of mass update scenario.

Kind regards,

Karri

Re: Dynamic Parallel Processing limitation

Former Member — Tue, 23 Jun 2015 18:28:18 GMT

Karri,

To answer your question on how is the Event Queue used in this case . . . The main workflow is started by a "material changed" event. This starts a process that creates a list of all the PO's and PR's affected. Using these 2 lists we start a subworkflow that basically gets invoked for each PO and/or PR found. These subworkflows are registered with the Event Queue. Using the Event Queue allows releasing of (currently) 50 workflows every 2 minutes. This feature also allows for monitoring and ease of reprocessing in case of errors. Who says that Dynamic Parallel processing is to be mainly used for approvals? I can use Dynamic Parallel processing to speed up a process, which was my goal in this case. Imagine doing 20 updates in parallel rather than in sequential mode, then multiply by several hundreds.

I welcome your suggestions, but in retrospect, I don't see how your solution (to process each item sequentially, outside the workflow) would be any better (how would you know which processes failed, how do you know if there are work processes out there available before releasing the next update, etc.). We did not have a single error until recently (so the volume was below the max limit of 999), and I can certainly modify the workflow logic/process to break down the list into manageable chunks.

Again, I thank you for your suggestions.

Regards,

Dan

Re: Dynamic Parallel Processing limitation

Former Member — Tue, 23 Jun 2015 21:42:58 GMT

Hi Dan,

your problem is not a one, which should be solved by a Workflow at all.

Do your updates directly in the receiver-type function module instead, replacing the SWW_WI_CREATE_VIA_EVENT_IBF with a copy of that one, holding your own code that updates the PO directly. No Workflow.

Best wishes

Florin

Re: Dynamic Parallel Processing limitation

SandySingh — Wed, 24 Jun 2015 00:03:31 GMT

Hello

If your are concerned about error handling during update failure, then you can create a custom application log to log errors and send a email (with attached application log details) to Distribution list for action. You can also include the email step/decision step for error handling in the workflow


I can certainly modify the workflow logic/process to break down the list into manageable chunks.

I think it is good idea to package the PO's and PR's related to material into groups for processing.

Regards

Sandy

Re: Dynamic Parallel Processing limitation

Former Member — Wed, 24 Jun 2015 02:02:00 GMT

Hi Dan,

>Who says that Dynamic Parallel processing is to be mainly used for approvals?

Well, I didn't. It was just an example. I have also used dynamic parallel processing for background steps/processes, but for a very limited amount of parallel "cases".

Basically I see only two ways out of this problem:

1) The "manageable chunks" option. (Now I understand your usage of the event queue, and in certain way it makes much sense.)

2) Redesigning your solution. I certainly would not use the workflow - or at least not in the way you are using it. You will run into these internal limitations of the WF engine (from my point of view WF was not designed to be some kind of mass processor of things - at least when it involves creating lots of work items under one parent work item). Sandy gave you some great hints about the error handling part, if that is one of your concerns.

One last question: Is it so that there could for example hundreds of material changed at the same time (by some background job, interface or whatever), and then each of these could cause a change to lots (hundreds) of PO/PRs. If this is the scenario, then this really gets interesting, and there is a great need for careful control of things...

Kind regards,

Karri

Re: Dynamic Parallel Processing limitation

pokrakam — Wed, 24 Jun 2015 18:01:52 GMT

Agree with overall sentiment that this is not what parallel processing is designed for. It's for process parallelisations, whereas the point of your design seems to be more for performance reasons.

My approach would be to submit one or more reports (can be triggered via event/WF), which will update all related orders. Any failures raise an event, which in turn triggers a manual correction/processing workflow.

Regards,

Mike

Re: Dynamic Parallel Processing limitation

Former Member — Wed, 24 Jun 2015 18:12:19 GMT

Thank you all for your valuable input.

I will go with the approach of submitting a batch job to do all the updates (using IDOCs). This way there will be room for individual restart ability in case an IDOC update fails.

My regards to all . . .

Dan