Strange package link behavior - data not saved!

[former_member186338]

I was requested to automate full reload of master data to one dimension in the model with full reload of transactional data on the daily basis.

Both master data and transactional data are available as csv files.

I decided to implement the following sequence (DM packages with below mentioned chains):

1. /CPMB/CLEAR to clear al data in the model

2. /CPMB/LIGHT_OPTIMIZE with Zero Elimination enabled

3. /CPMB/IMPORT_IOBJ_MASTER with Overwrite and Clear All - only single dummy member imported.

4. /CPMB/IMPORT_MASTER from csv file with dimension master data.

5. /CPMB/IMPORT from csv file with transactional data.

Successfully tested all individual DM packages (on BPC 800 and 801).

Then I decided to create a package link to combine all 5 steps.

And the last step during the package link execution doesn't store transactional data in the model!

The log shows Success, correct number of records submitted...

But no values on the report and in the cube using RSA1.

After some investigation I managed to found the missing records in the Fact table, but with incorrect SID for the dimension that was reloaded in the previous step. The SID was equal to the previous SID of the member (before dimension master data reload).

Any idea to insert some step before loading transaction data to refresh SID's?

P.S. For sure I can create 2 package links with some time interval between (one for master data the second for transactional data). But looking for more elegant solution...

[former_member186338 · Accepted solution]

Closing old questions. Issue is solved with some SP's.

[Former Member]

Hi Vadim,

Is this issue always on same Dimension or on different? If it always on same can you check if that number range is buffered? Sometimes it's done to speed up loads.

Also, does you cube have DIM tables? If it has, can you check if after cube is emptied do those DIM tables still exist or they are gone?

Regards,

Gersh

[former_member186338]

Hi Gersh,

I have 2 different systems (800 and 801) with different cubes with different dimensions but with the same logic in package links. The issue is 100% reproducible on both (on both systems only one dimension is used).

If I remove CLEAR first step and manually launch CLEAR package then launch package link - no issues!!

If I manually run all 5 DM packages one after one - no issues!

"can you check if that number range is buffered" - how?

P.S. The issue is somehow related to Package Link execution...

[Former Member]

Hi Vadim,

Difference could be that when you run it manually you kick off different programs. When you run it in Package link it runs all inside one program. So, perhaps some cleanup that is done when you go out of the program is not happening in this case.

To check Number ranges you can use t/a SNRO.

-Gersh

[former_member186338]

I can see only "No buffering" on SIDs Characteristic /CPMB/NGDCP1Q - dimension with issue...

[former_member186338]

"Also, does you cube have DIM tables?" I have tested the package link on a very simple cube with 5 dimensions and small number of members in each dimension...

[Former Member]

Can you check if splitting this into 2 processes will fix the issue? If you can split your PL into 2 and run one after another without waiting? Like with a process Chain. Would this fix the issue?

If it fixes then we can try to find out how to split it in most natural way. If it doesn't, then something else is an issue:(.

[former_member186338]

Hi Gersh,

What do you exactly mean by "splitting this into 2 processes"?

Vadim

[Former Member]

Hi Vadim,

I mean creating 2 PLs: one with first 3 steps and another with the last 2 steps. Create variants for each of them using UJD_... program. Create Process Chain where 1st step calls the program with one variant and 2nd step call same program with the other variant.

Run that PC.

Regards,

Gersh

[former_member186338]

OK, PL = Package Link

And variant of UJD_TEST_PACKAGE_LINK

[former_member186338]

Yes, it's working this way:

PL Clear run UJD_TEST_PACKAGE_LINK with the PL with first 3 DM packages (CLEAR, Light Opt, Load MD from BW)

PL Load run UJD_TEST_PACKAGE_LINK with the PL with last 2 DM packages (Load MD from File, Load TD from File).

But still, I have no idea why the original package link have this bug!

[gregor_dieckmann]

Hi Vadim,

as mentioned above I think this is a buffering problem not of a BPC dimension but of a BW dimension in the BPC generated InfoCube. In BW - in general - more than one BW characteristic is contained in a BW dimension; in BPC generated InfoCubes there is just one characteristic in a BW dimension (except may be the 13. dimension). But in general, writing delta records (characteristic values based, in short CHAVLs) BW has to read the SIDs of the CHAVLs and to determine the BW dimension ID per SID combination.

Function module RSDRI_CUBE_WRITE_PACKAGE used in BPC to write data has a SID as well as a DIMID buffering and the question is whether it is technically ok to combine deletion of InfoCube data + deletion of master data + creation of master data + creation of new transaction data in one ABAP internal mode. In BW deletion of transaction data is usually done in RSA1 in one transaction. So deletion of transaction data, master data and creation of master data, transaction data is not in one ABAP internal mode. As a result, buffering of DIMIDs or SIDs is no problem since different ABAP mode don't share data.

So my theory is that DIMID buffers (and maybe SID buffers) are not invalidated in your case. You might debug RSDRI_CUBE_WRITE_PACKAGE to verify this. The real stuff happens in a generated program, you get the name of the program using function module RSDD_INFOCUBE_WRITEPROG_GEN in test mode with the BPC generated InfoCube name. In this report you can see some hashed tables to buffer SIDs and DIMIDs. In class CL_RSDD_CUBE_WRITER, method WRITE you can also find method CHECK_KEEP_DIMCACHE that might be of interest.

If my theory is correct then it is still unclear whether BPC has to adjust the way BW services are called or whether BW has to be prepared to run all the stuff in one ABAP internal mode. For this you should open a customer ticket.

Regards,

Gregor

[Former Member]

Hi Vadim,

Can you take a look at Technical setting of SID Dimension that causing the issue: /B28/SNGDCP1Q? It should show if that tables allows buffering. I'd think that it does.

Can you disable it and run your process again?

Regards,

Gersh

[former_member186338]

Hi Gregor,

From the BPC point of view I am doing nothing wrong combining all processes in a single package link. You theory is close to my understanding of things. May be I will find time to perform debugging.

Looks like:

1. Cash is created when CLEAR package write delta records to have zero as a result.

... Manipulations with master data... New SID's etc...

5. The same old cash is used for import transactional data. No errors generated. Data disappears

Any idea how to invalidate this cash? May be some abap program (to run inside package link as separate DM package)?

Vadim

[former_member186338]

Tested (switched off buffering!):

No result, same issue

[former_member186338]

As a temporary working solution I configured 2 package links:

And a custom chain:

Where:

PL Clear TD: UJD_TEST_PACKAGE_LINK with variant to run first package link

PL Clear MD Load MD and TD: UJD_TEST_PACKAGE_LINK with variant to run second package link.

[former_member186338]

Some screenshots to illustrate the issue!

Package link:

Log of the last package:

SID table of USER22 dimension BEFORE package link run:

SID table of USER22 dimension AFTER package link run:

And Fact table AFTER package link run:

SID 153 is from previous state of SID table! Has to be 159!

For sure no data in the report

[gregor_dieckmann]

Hi Vadim,

this looks like a BW dimension and/or SID caching problem. Have you checked BW notes, e.g. 2035451?

Regards,

Gregor

[former_member186338]

Currently I have no access to SAP notes - Changing S'iD

Can you provide the text?

[gregor_dieckmann]

The note is about:

Loading allowed in DELTA LOAD InfoCube; always writes 0 to fact table as dimension key

and a fix is included in BW 7.40, SP09; this seems to be BW 7.40-specific.

The BPC generated BW InfoCube has to have the flag DELTA_LOAD = X what you can check in table transaction RSDCUBE.

[former_member186338]

I have this issue with:

And with:

P.S. This setting?

[Former Member]

Hi Vadim,

Have you tried using UJD_TEST_PACKAGE_LINK to run the package link from BW???

Which  version are you using? 10 or 10.1?

Regards,

JP

[former_member186338]

Hi JP,

But what is the idea behind running package link with UJD_TEST_PACKAGE_LINK???

By the way in the original post I have "BPC 800 and 801"

Vadim

[Former Member]

Hi Vadim,

The idea was to check whether there is a lag in updating data in tables when you run the link from BPC vs BW directly itself. Because in my case, in one of my projects I had done the automation of loading the data from ECC to BW to BPC by creating a process chain. While loading from BW to BPC I was using DMP link to load the master as well as transaction from 3 different cubes... and it ran successfully. It was BPC 10 SP 09. I was using Light optimization in the end of all DMP's in the link!

Sorry, I dont have the knowledge about which version 800 and 801 refers too...

Regards,

JP

[former_member186338]

I will test...

"Sorry, I dont have the knowledge about which version 800 and 801 refers too..." -

Absolutely basic:

800 and 801 - BPC 10 (on different BW)

810 - 10.1

[Former Member]

Oh ok....Thanks Vadim...

[former_member186338]

Just tested with UJD_TEST_PACKAGE_LINK

Same result

[Former Member]

Why you don't try my solution?

[former_member186338]

Not very elegant

And first of all I am trying to understand the reason of the bug! And it's definitely a bug...

[Former Member]

What? You add an extra step in your package link, how is that not elegant? There is no bug, you need to refresh the cache after you update the master data.

Andy

[Former Member]

. Have you checked solution given in this thread....https://scn.sap.com/thread/3879119

JP

[former_member186338]

"There is no bug, you need to refresh the cache after you update the master data." - sorry, but this statement was never documented! And I never see the bug like this in this forum...

And by the way if I remove first step - CLEAR data from the link and clear data manually the package link will execute correctly

[former_member186338]

No errors in RSRV test!

Actually I have checked SID's manually in my first investigation, RSRV confirmed the same...

RSRV is not checking contents of Fact table...

[Former Member]

Hmm...You said when you clear the data manually then the package link is executing correctly. What if we execute Clear and Light optimization in one package and all other 3 steps in another package, schedule this package after a gap of lets say 2 or 3 mins after the 1st link!

Regards,

JP

[former_member186338]

This idea was already mentioned it the original post But I don't like it due to unpredictable interval

[Former Member]

Your was to separate master and transaction....mine was to separate clear and data loading if clear was making any problem.... Anyways.....Then I guess go to SAP Support....no other option left(atleast with me).

Regards,

JP

[former_member186338]

"Your was to separate master and transaction....mine was to separate clear and data loading if clear was making any problem" Sorry, but it's the same approach (and both will work somehow)! In both cases we have to set some unpredictable time interval to complete the first package link!

[former_member186338]

I also tried to import transactional data twice with the same steps 5 and 6. Same issue!

Can you test this scenario in your system?

[Former Member]

Hi Vadim,

As requested by you I checked the scenario in my system.

1.) Create a package link to upload master(GL ~ 150) as well as transaction data(~ 2400). The  package link ran successfully without any errors.

2.) But if I include Clear package with the above 2 packages in the package link, the package link Fails and it fails the Clear package itself! But if I execute the Clear package only outside the link it works...

This is really strange now! My BPC n BW versions. Definitely a bug!

[former_member186338]

There is something wrong with answer prompt for clear package in your package link. Never failed in my case!

And to reproduce my scenario you have to create all 5 steps! Including 2 step of master data load (first from BW - single record to clear master data in dimension, second - from file to reload master data). Light Optimize with zero elimination is also required to be able to delete master data!

[Former Member]

with this package /CPMB/IMPORT_IOBJ_MASTER I only get Overrite option and not Overwrite and Clear all...Do I proceed with it?

[former_member186338]

Sorry, but you have to upgrade DM instructions using UJS_ACTIVATE_CONTENT...

The current advanced script for /CPMB/IMPORT_IOBJ_MASTER is:

PROMPT(MASTERSELECTION, %INFOOBJECT%, %SELECTION%, "Please select the infoobject and the selection",,)

PROMPT(RADIOBUTTON,%WRITEMODE%,"Write Mode",2,{"Overwrite","Update"},{"1","2"})

PROMPT(RADIOBUTTON,%FORMAT%,"Choose format.",1,{"Internal Format","External Format"},{"1","0"})

PROMPT(TRANSFORMATION,%TRANSFORMATION%,"Transformation file:",,,Import.xls)

PROMPT(DIMENSIONNAME,%DIMNAME%,"Dimension name:",,,%DIMS%)

PROMPT(RADIOBUTTON,%CLEAR_ALL%,"Clear all members for overwrite mode before loading?",0,{"No","Yes"},{"0","1"})

PROMPT(RADIOBUTTON,%FILTERMD%,"IF Filter Master Data",2,{"Yes,Filter Master Data Before Importing","No,Do Not Filter Master Data Before Importing"},{"1","2"})

INFO(%TEMPNO1%,%INCREASENO%)

INFO(%TEMPNO2%,%INCREASENO%)

INFO(%TEMPNO3%,%INCREASENO%)

INFO(%TEMPNO4%,%INCREASENO%)

INFO(%TEMPNO5%,%INCREASENO%)

INFO(%TEMPNO6%,%INCREASENO%)

INFO(%TEMPNO7%,%INCREASENO%)

TASK(/CPMB/BW_IOBJ_MASTER_SOURCE,OUTPUTNO,%TEMPNO1%)

TASK(/CPMB/BW_IOBJ_MASTER_SOURCE,MASTERDATA_SELECTION,%SELECTION%)

TASK(/CPMB/BW_IOBJ_MASTER_SOURCE,INFOOBJECT,%INFOOBJECT%)

TASK(/CPMB/BW_IOBJ_MASTER_SOURCE,MASTERDATA_FORMAT,%FORMAT%)

TASK(/CPMB/BW_IOBJ_TEXT_SOURCE,OUTPUTNO,%TEMPNO2%)

TASK(/CPMB/BW_IOBJ_TEXT_SOURCE,MASTERDATA_SELECTION,%SELECTION%)

TASK(/CPMB/BW_IOBJ_TEXT_SOURCE,INFOOBJECT,%INFOOBJECT%)

TASK(/CPMB/BW_IOBJ_TEXT_SOURCE,MASTERDATA_FORMAT,%FORMAT%)

TASK(/CPMB/MERGE_IOBJ_MD_DATA,INPUTNO,%TEMPNO1%)

TASK(/CPMB/MERGE_IOBJ_MD_DATA,INPUTNO2,%TEMPNO2%)

TASK(/CPMB/MERGE_IOBJ_MD_DATA,MASTERDATA_SELECTION,%SELECTION%)

TASK(/CPMB/MERGE_IOBJ_MD_DATA,INFOOBJECT,%INFOOBJECT%)

TASK(/CPMB/MERGE_IOBJ_MD_DATA,OUTPUTNO,%TEMPNO3%)

TASK(/CPMB/IOBJ_SOURCE_MD_CONVERT,FORMULA_FILE_NO,%TEMPNO7%)

TASK(/CPMB/IOBJ_SOURCE_MD_CONVERT,TRANSFORMATIONFILEPATH,%TRANSFORMATION%)

TASK(/CPMB/IOBJ_SOURCE_MD_CONVERT,SUSER,%USER%)

TASK(/CPMB/IOBJ_SOURCE_MD_CONVERT,SAPPSET,%APPSET%)

TASK(/CPMB/IOBJ_SOURCE_MD_CONVERT,SAPP,%APP%)

TASK(/CPMB/IOBJ_SOURCE_MD_CONVERT,DIMNAME,%DIMNAME%)

TASK(/CPMB/IOBJ_SOURCE_MD_CONVERT,OUTPUTNO,%TEMPNO4%)

TASK(/CPMB/SPLIT_IOBJ_MD_TXT_DATA,INPUTNO,%TEMPNO4%)

TASK(/CPMB/SPLIT_IOBJ_MD_TXT_DATA,OUTPUTNO,%TEMPNO5%)

TASK(/CPMB/SPLIT_IOBJ_MD_TXT_DATA,OUTPUTNO2,%TEMPNO6%)

TASK(/CPMB/IOBJ_SOURCE_MD_CONVERT,INPUTNO,%TEMPNO3%)

TASK(/CPMB/BPC_MASTER_DATA_TARGET,INPUTNO,%TEMPNO5%)

TASK(/CPMB/BPC_MASTER_DATA_TARGET,DIMNAME,%DIMNAME%)

TASK(/CPMB/BPC_MASTER_DATA_TARGET,WRITEMODE,%WRITEMODE%)

TASK(/CPMB/BPC_MASTER_DATA_TARGET,FORMULA_FILE_NO,%TEMPNO7%)

TASK(/CPMB/BPC_MASTER_DATA_TARGET,CLEAR_ALL,%CLEAR_ALL%)

TASK(/CPMB/BPC_MASTER_DATA_TARGET,FILTERMDMODE,%FILTERMD%)

TASK(/CPMB/BPC_TXT_DATA_TARGET,INPUTNO,%TEMPNO6%)

TASK(/CPMB/BPC_TXT_DATA_TARGET,DIMNAME,%DIMNAME%)

[former_member186338]

P.S. But may be your SP11 for 800 is too old!

[Former Member]

In that case Vadim...I am sorry I cant help . I am not in a position to upgrade the system to latest SP and check!

[former_member186338]

"But may be your SP11 for 800 is too old!" - I am not sure... at what SP the Clear All was introduced. Just try UJS_ACTIVATE_CONTENT and create new package with this chain!

[Former Member]

I just created a new package after updating the DM instructions. But get error as after running :

Cannot configure parameter CLEAR_ALL in task /CPMB/BPC_MASTER_DATA_TARGET.

Got the Clear all members task in DM instructions but no use. Anything else is required?

[former_member186338]

Just go to RSPC, edit chain and add:

Or update BW content in UJS_ACTIVATE_CONTENT

[Former Member]

Hi Vadim,

You need to add an extra step in your package chain, the extra step would be triggering process chain in BW, and that process chain need to include UJA_REFRESH_DIM_CACHE step for the dimension.

Andy

[former_member186338]

For testing I have created BPC process chain with run abap program - UJA_REFRESH_DIM_CACHE. Created DM package with this chain and add it in the package link before transactional data load. No success, same issue!

[lucas_costa3]

Weird behavior. Is this model hana optimised? I had some issues with data sync between hana and bw.

Maybe include the optimisation after master data loading, can that put the SIDs in order before importing the TD.

Cheers

[former_member186338]

Hi Lucas,

Both systems are running without HANA.

And yes, the first thing I tested was to include extra Light Optimize step before step 5 - no luck!

Vadim

[lucas_costa3]

Right... Well, agree with you: is definitely a bug.

Now, to at least temporarily fix this the only thing would come come to my mind except the idea that you've already tried, is to include a step to process the dimension you're updating via abap, but not "elegant". If you decide to use this check interface if_uja_member_manager method process.  

Cheers,

Lucas