Data Services to capture only changed data

[Former Member]

Hi Experts,

I have below basic DS job that combines multiple tables into single target table.

DF_Cycles loads million of records from source to target and takes long time to run...How can i tell DS to load only changed data based on a OID?. Below sample data for your reference.

And the seconds DF have bigger tables with about 20 million records, which has join on OID and every execution of the job is going through all the records to see if the OID from DF_Cycle target exists in Activities SQL. Pls see below.

How can i tell DS to compare only changed from DF_Cycles and make the execution faster. Below screenshot for your reference.

Thanks in advance for your help.

[denise_meyer]

for DF_Cycles - use the table_comparison transform - see the documentation.

https://help.sap.com/viewer/e54136ab6a4a43e6a370265bf0a2d744/4.2.13/en-US/5770ef146d6d1014b3fc9283b0...

You can redesign the second dataflow to not use SQL transforms and potentially pushdown the sql to join all these tables and filter them to the database to make the processing faster. Or you can try adding a data transfer transform to have this pushed down as well to the database

here is the performance optimization guide for your reference as well

https://help.sap.com/viewer/5d0c467218d5489791d04ced984a8f89/4.2.13/en-US