Processing large data on Python3 Operator in SAP Data Intelligence

[former_member759762]

Hi,

I'm trying to process the data on Python3 Operator but the data have many rows (about 200 thousand rows).

On Python3 Operators the data was split to some chunks.

How to combine all the chunks or iterate and processing the data by chunk on Python3 Operator?

I already set the batch size but the data still split to the same number of chunks.

Thank you.

Regards,

Agus

[former_member759762]

Finally found the solution,

Based on this reference: How to collect output of LIST operator in batch | SAP Community

Here is how to append all the chunk in batch.

import pandas as pd
import io

df = pd.DataFrame()

def on_input(data):
    global df
    
    body = io.StringIO(data.body)
    dfchunks = pd.read_csv(body, sep=",", header=0)
    df = df.append(dfchunks, ignore_index=True)
    
    if data.attributes['message.lastBatch']:
        api.send("output", str(df.shape))

api.set_port_callback("input", on_input)

Thank you,

Regards,