Showing results for 
Search instead for 
Did you mean: 

Can we remove duplicate records at the runtime in dataflow?

0 Kudos

Hello Experts,

Is it possible to remove the duplicate records at runtime when dataflow is executing on list of records.

e.g. An excel file of 10 records out of which 5 records are unique and remaining are duplicates. When we run the dataflow on same file, dataflow processes 10 records instead of 5 unique records. Which dataflow step can filter these duplicate records and process only unique records?




View Entire Topic
Active Contributor
0 Kudos

Hello @grohitg238,

I recommend you do the easy DEDUPLICATION in Excel itself, instead of giving extra load to Dataflow to focus on importing the right contacts, regardless of the volume of data in the EXCEL file.

We recently followed similar approach for millions of data.

I hope this answers your query.

Best Regards,
Kunal Bansal | SAP CX Solution Architect 


Yes Kunal, Partially. We want automate it the process instead of manual. Thanks for your valuable suggestion, much appreciated.