Handling Threading in Python3 Operator of SAP Data Intelligence

[former_member804506]

Hi,

I am getting high frequency of Data Batches at the input port of Python3 Operator and every time the data comes, the corresponding port_callback function kicks off. I am suspecting that because of high frequency of data, Maybe the data is getting missed because of losing some of the batches in between.

Can i know how do we handle the threading in Python3 operator and any specific Code snippet if possible?

Looking forward to get support from SAP DI and Python3 Community. 🙂

Thanks

[DeLo]

Hello Saurabh,

How do you realize some batches are missing? I am asking because there might be an issue related to the send() function that you use to pipe your data out of the python operator. The send() function is asynchronous and doesn't copy the data being sent. This means that the next input might overwrite the data being sent before you actually send them. The workaround is to deepcopy the data before you send them out.

Not sure this applies to your case. If not, please add more details: a graph snapshot or the python code might help.

Regards.

[former_member804506]

Hi,

When you say "To DEEPCOPY" the data, what does it mean? To stage it somewhere? as far as staging is concerned, Staging data (even temporary) is not allowed in my usecase.

Second question: is it the same case for port_callback() function? if continuous, high frequency of batch data is coming on a particular port, will the current batch overwrite on previous batch?

Regards

[DeLo]

deepcopy is a python method to make sure you pass a variable to a method by copying it and not by referencing. The copy is only temporary and won't stage anything: it is dropped when the function returns. More details here: https://docs.python.org/3/library/copy.html

set_port_callback() does NOT behave the same.

[Cocquerel]

You can refer to the following blog
https://blogs.sap.com/2022/03/17/comprehensive-guide-to-parallel-processing-in-sap-data-intelligence...

[former_member804506]

Hi Michael,

Thanks for your inputs.

Regards