Data Intelligence: How to remove unnecessary columns from csv data?

[MioYasutake]

Hi, I'm new to Data Intelligence and would appreciate your advice.

I'm trying to extract tale data form S/4HANA using STL connector. After that the data is converted into csv and transferred to HANA Cloud. The image below is the pipeline created.

After ABAP converter, data has the following structure. Notice that table name and a flag (U for update and D for delete) are appended.

As a result, upsert to HANA fails with the following error.

Graph failure: column count does not match incoming data: expected 16; received 18 (["400" "ZT-1000" "PR" "NotebooksBCD" "" "0100000046" "EUR" "100.00" "" "RISC989_FI" "2021-11-29" "04:07:28.000" "" "9999-99-99" "00:00:00.000" "" "ZDEMO_PRODUCT" ""]); this may be caused by setting the wrong record delimiter

I think I need to remove those additional columns before doing upsert. But how can I do that? Is there a simple way than writing a script?

Best regards,

Mio

[ankitsharma_71 · Accepted solution]

Hi Mio,

Two ways you can achieve this:

First approach:

a. Use python operator in between ABAP converter and SAP Hana client operator.

b. In Python operator , write a script to load the data into dataframe.

c. Then you can remove the unwanted columns from dataframe and send the output from dataframe to SAP Hana client operator.

Second approach:

a. Use Gen2 Read Data from SAP operator and select the source.

b. Connect Gen2 operator with Data Transform operator and mapped the required columns in target.

c. Connect Data Transform operator either with Write Hana Table/ Table producer and load the data in target table.

Thanks,

Ankit

[MioYasutake · Accepted solution]

Hi ansharm7,

Thank you for your response. I followed the first approach and it worked.

Regarding the second approach, our backend was not ready to use Gen2 operators, so I chose the first option.