CDS view with Change Data Capture(CDC) for Replication Flow – Part 1

[SayaniGhosal]

SAP Datasphere is the next generation of SAP Data Warehouse Cloud and delivers a unified experience for data integration, data cataloguing, semantic modelling, data warehousing, data federation, and data virtualization. Data Integration is an essential topic in a Business Data Fabric like SAP Datasphere and Replication Flow is the cornerstone to fuel SAP Datasphere with data, especially from SAP ABAP sources.

Based on the different data sources for a Replication Flow, you have different options for replicating the data in SAP Datasphere. If your source system is an SAP S/4HANA system offering the needed data by means of CDS views, directly connecting SAP Data Intelligence to this source system and using the respective CDS views is the best approach.

In this blog, we will try to delve into the details of CDS configuration to make it Delta Load compatible for a Replication Flow based extraction:

In Replication Flow settings, we have below load types available:

1. Initial – If you select this option, full data retrieved by the CDS view will be transferred to target every time the Replication Flow is executed. 
2. Initial and Delta – If you select this option, full data retrieved by the CDS view will be transferred to target for the first run, post that at scheduled intervals the Replication Flow will transfer delta records. For Initial and Delta setting, RF uses CDC delta.

The ODP framework for CDS extraction provides two options for delta handling:

• Generic Timestamp / Date based Delta (Not supported by SAP Datasphere 2024.23 Replication Flow )
• Change Data Capture (CDC) Delta

Starting with SAP S/4HANA Cloud 1905 and SAP S/4HANA 1909 FPS01 (on-premise) additional delta capability of Change Data Capture is available. This delta option captures changes to the application tables based on data base triggers and enables the ODP framework to just provision these newly created/changed/deleted records to the consumers.

CDS Annotations for enabling Replication into Datasphere

A. Annotation for Data Extraction

To make the data from an ABAP CDS view available in SAP Data Intelligence Cloud(Replication engine), you need to specify annotations for the view in the ABAP Development Tool.

To load data(initial only or initial with delta) from a custom CDS view, add the following annotation to the view:

• Analytics.dataExtraction.enabled - This annotation exposes the CDS view for Consumption by a Replication Flow. For both Initial and Initial with Delta Load types this annotation is mandatory.

B. Annotation for CDC in simple CDS view

As mentioned before, the Change Data Capture recording mechanism uses database triggers to record any changes to the tables that belong to an ABAP CDS view. To do this, the key fields of all underlying tables need to be mapped to the fields of the CDS view.

For simple CDS views such as CDS projection views, this can be done automatically by using the following annotation:

• Analytics.dataExtraction.delta.changeDataCapture.automatic – This annotation will enable the use of CDC (change data capture) delta mechanism based on data base triggers. Manual key mapping in ADT is not required as we are using a very simple CDS projection view. In case of automatic mapping system relies on the key field of the main table used in CDS.

Let’s take an example of a standard view and see how it works : 

CDS  view : I_SUPPLIER

Although this CDS view seems complex, all data retrieval is through association(no joins used). The main table here is LFA1. Hence CDC(delta capture) only needs to track changes on LFA1, so the simple annotation is used for automatic CDC.

C. Annotation for CDC in complex CDS view

For more complex CDS views that use joins, instead of using CDC automatic annotation we do the table key mapping manually in CDS. You will be using the following annotation to do this:

• Analytics.dataExtraction.delta.changeDataCapture.mapping

What is table key mapping?  During delta capture, SAP stores the delta entries in internal logging tables and the key fields of these logging tables will be decided based on the key fields of the main tables(joins) used in CDS.

Let’s take an example of a standard view and see how it works : 

CDS view : I_WBSELEMENTDATA

Screenshot I :

Screenshot II :

As per Screenshot I, we have extraction enabled, CDC mapping. Let’s have a look at the CDS design in Screenshot II to understand the mapping.

Using this mapping mentioned in the annotation, Change Data Capture framework performs logging of the delta records.

Once CDS view is created, it can be used in Replication Flow to populate data in SAP Datasphere.

SAP Datasphere has a link with the prerequisite SAP notes  https://help.sap.com/docs/SUPPORT_CONTENT/datasphere/4457678727.html?q=pre%20requisite

Hope you find this blog helpful! I will write a subsequent blog to elaborate the CDC logging in more details.

[Cocquerel]

What about the CDS annotation DataIntegration.deltaReplication.intended  ? Does it also generate database triggers ? What's the difference with the Analytics annotations you have described and why there are 2 different CDS annotations using 2 different framework for the same capability (CDS delta replication) ?

[albertosimeoni]

@Cocquerel 

You find the mother of all easter eggs, from the link you provide it seems just another annotation without any real effect (like VDM.type: #BASIC)

[SayaniGhosal]

Hi @Cocquerel , 

@albertosimeoni has already addressed the first part of your question.

The 2 different CDC annotation A and B in the blog is dependent on the complexity of CDS view. If it's a simple projection view you use the automatic annotation (A), if it's a complex CDS view it's the mapping annotation(B). Both the annotations use the same framework(CDC). 

Best Regards,

Sayani 

[c_baker]

This is not SAP Replication Server.  Please remove the tag.

Thanks,

Chris

[BenedictV]

hi @SayaniGhosal ,

Does the table key mapping ensure any changes to the left table fields are also captured as part of delta runs?

-Benedict

[SayaniGhosal]

Hi @BenedictV ,

Yes, when there's a join in CDS view, we have to use the Table Mapping for CDC so that the changes to the left table are captured as part of delta.

Regards,

Sayani

[Maha-Dev]

Is Delta capture enabled for the Local table? How does the delta mechanism work in the delta table (tableXYZ_delta)?

[Balu483]

@SayaniGhosal ,

How to deal with the below scenario?

we had setup CDS Change data capture and deployed in production. after few months of deployment due to some connectivity issue or some other reasons some data was not reached to target application in this cases how can we able to reprocess manually by providing some input (for example Creation date ) and execute?

[SayaniGhosal]

@Balu483 : From a replication flow perspective, if there's a mismatch in the delta entries, you would need to stop and restart the Replication Flow run which will load full data in the target once more.