Capture updates on an ECC table

[Noor534]

Hi all,

Appreciate if you could give feedback on the below data quality related problem statement and solution idea -

The problem at hand is data updates in SAP ECC (source) are not posted by SLT to Databricks (target) at times, for few tables, at least 5 are caught so far based on incidents from users. In other words, the CDC technology from SAP i.e. SLT is missing updates, not sure why. This is a separate topic to deal with.
To ensure integrity of data in Databricks, doing column-by-column and row-by-row reconciliation between SAP ECC and Databricks is tedious and may not be sustainable, as we have several large tables in scope. To avoid reconciliation, I was thinking about this option  -

just extracting the number of updates per day on each table, and verify in Databricks if the number is same. Example: currently. we had updates on materials (MARA) missed in Databricks. So, if I can get the number of updates on MARA for a particular day, that would be great.

Please let me know the best way to capture the number of updates. Note - we have around 10000+ tables. Also, CDPOS & CDHDR tables that could contain this information are stale at times, due to their huge size.

Best Regards,

Noor.

[MKreitlein]

Hello @Noor534 

before you invest a lot of work in what you describe... have you checked that your system is up-to-date?

There is the following KBA: https://me.sap.com/notes/3016862/ 

I think you need program: CNV_NOTE_ANALYZER_SLT

.. to check whether there updates for CDC.

Only if you have all available notes in the system... then you could maybe check if you find the changes via CDHDR and CDPOS tables?

BR, Martin 

[Noor534]

Hello @MKreitlein, Thanks for the reply. I am unable to open the notes.

[Noor534]

Hello @MKreitlein, Thanks for the reply. I am able to download the notes now.

[Noor534]

Hello @MKreitlein, Just a quick thought - even if there exist all notes in SLT, assuming we like to build data observability around data updates/inserts/deletes between SAP ECC and Databricks, can you please guide how I could capture the metrics on SAP ECC side? Appreciate the support.

[MKreitlein]

Hello @Noor534 ... I think your approach requires high effort for very small benefits.

I don't know if you have a delay of 1 hour as of now, between ERP and Databricks... when (in time) do you think you can compare the numbers to find a 100% match?

I don't know about any statistics tables in ERP... so you would have to count changes on your own, if this works at all with the change tables.

Just focus on the note completeness first 😉