SAC data refresh - Incremental load and Past data refresh at same time.

[#1]

Dear Team,

Greetings for the day!

We have a business requirement where we have to load data into SAC models with following conditions (explained using an example)

Conditions:

Please consider current day as 23rd March 2021. The connection used is "import connection" with SAP BW.

User wants to extract new data from source system (SAP BW) for 22nd March 2021 (incremental load)
User also had made changes in source system for date 01st March 2021 and wants it to get reflected in SAC (update refresh)

Volume of data for single day increment - 10000 rows (can be handled by SAC, as it is much within its thresholds)

Volume of data for a month update refresh - 3,00,000 rows (surpasses SAC threshold limits ).

Is there any way of handling incremental load and update query in a single SAC data model.

Thanks in advance,

Pratik

[DebjitSingha]

Hello pratik1612,

What you need is an incremental date field at your source (SAP BW). It can be "Created On", "Posting Date", "Refresh Date", "Entry Date" as long as it account for daily data. If you don't have any such date field then, consider adding one. Say you add a new date field (Refresh Date) into your cube/ aDSO and write a simple routine to consider system date as entry. Whenever data loads into your infoprovider, this date field gets update.

Now historical data need to be processed manually first. Say couple of days at a time. Based on filter over "Refresh Date" till we reach March 22nd.

Similar to below example:

SAC lets you consider date fields as incremental fields. In your case, you can consider Refresh Date. But before that add a filter entry as Refresh Date >= 03/22/2021 and on the next screen click on "Incremental Load

Note: You can decide the import method as per your requirement, although as your are considering data incremental based on Refresh Date (or similar fields), you can pick Import Method as Update.

Viewers/ readers - If you find above information helpful, feel free to up-vote (arrows on left side).

- Debjit

[Former Member]

Hi Debjit,

I have already used incremental load feature. The issue is user wants both incremental load and past data update at same time.

Best regards,

Pratik

[DebjitSingha]

Hello Pratik,

You need a custom "Refresh Date" field in your BW. This field will get updated independent of your existing date filed. Say source data changes (flat file) for the entire month (couple of data points in past). BW transformation will flag new and changed rows with system date.
You do not have to reload entire March in this case. Only changed and new records. Chances are the total count will be way less.

Example:

Say a FI document is created on 10th March and cleared on March 23rd. So if you refresh SAC based on created on date, then you are in a situation where you are only picking this document on 10th only. Changed record (on 23rd) are not considered. Onlyway to capture changes would be based on "changed on date" or "clearing date". So, you have to setup one job based on Creation Date/ posting date and the other based on Changed On Date / Clearing Date.

If you setup "Refresh Date" column in BW, then your setup will flag this one document twice on 10th and again on 23rd. SAC will be based on Refresh Date and not on Created On field. This way you changed records get captured based on "Refresh Date" field.

Let me know if you have any question.

Thanks,

Debjit

[Abd_Am_K]

Hi pratik1612

Adding to Debjit's answer. Here is the help link to the sizing. You can also check this answer for reference. Regarding your updating of query, it depends. You can change the filter in your query as often as you want. However, if you have a new dimension then AFAIK you will need to create a new Model (or recreate the existing one) to add any new dimensions added.

Thanks,

Amer