Exporting data clusters with type version

[Ruediger_Plantiko]

Hi all,

let's assume we are saving some ABAP data as a cluster to database using the IMPORT TO ... functionality, e.g.

EXPORT VBAK FROM LS_VBAK VBAP FROM LT_VBAP  TO DATABASE INDX(QT) ID 'TEST'

Some days later, the data can be imported

IMPORT VBAK TO LS_VBAK VBAP TO LT_VBAP FROM DATABASE INDX(QT) ID 'TEST'.

Some months or years later, however, the IMPORT may crash: Since it is the most normal thing in the world that ABAP types are extended, some new fields may have been added to the structures VBAP or VBAK in the meantime.

The data are not lost, however: Using method CL_ABAP_EXPIMP_UTILITIES=>DBUF_IMPORT_CREATE_DATA, they can be recovered from an XSTRING. This will create data objects apt to the content of the buffer. But the component names are lost - they get auto-generated names like COMP00001, COMP00002 etc., replacing the original names MANDT, VBELN, etc.

So a natural question is how to save the type info ( = metadata) for the extracted data together with the data themselves:

EXPORT TYPES FROM LT_TYPES VBAK FROM LS_VBAK VBAP FROM LT_VBAP TO DATABASE INDX(QT) ID 'TEST'.

The table LT_TYPES should contain the meta type info for all exported data. For structures, this could be a DDFIELDS-like table containing the component information. For tables, additionally the table kind, key uniqueness and key components should be saved.

Actually, LT_TYPES should contain persistent versions of CL_ABAP_STRUCTDESCR, CL_ABAP_TABLEDESCR, etc. But it seems there is no serialization provided for the RTTI type info classes.

(In an optimized version, the type info could be stored in a separate cluster, and being referenced by a version number only in the data cluster, for efficiency).

In the import step, the LT_TYPES could be imported first, and then instances for these historical data types could be created as containers for the real data import (here, I am inventing a class zcl_abap_expimp_utilities):

IMPORT TYPES TO LT_TYPES FROM DATABASE INDX(QT) ID 'TEST'.
DATA(LO_TYPES) = ZCL_ABAP_EXPIMP_UITLITIES=>CREATE_TYPE_INFOS ( LT_TYPES ).
assign lo_types->data_object('VBAK')->* to <LS_VBAK>.
assign lo_types->data_object('VBAP')->* to <LT_VBAP>.
IMPORT VBAK TO <LS_VBAK> VBAP TO <LT_VBAP> FROM DATABASE INDX(QT) ID 'TEST'.

Now the data can be recovered with their historical types (i.e. the types they had when the export statement was performed) and processed further.

For example, structures and table-lines could be mixed into the current versions using MOVE-CORRESPONDING, and so on.

My question: Is there any support from the standard for this functionality: Exporting data clusters with type version?

Regards,

Rüdiger

[matt]

I don't know if there's anything standard, but at the conclusion of this discussion, please could you write a blog with this information. It's not something I'd thought about before.

[Ruediger_Plantiko]

Hi Matthew,

I don't know if there's anything standard, but at the conclusion of this discussion, please could you write a blog with this information.

from the length of the question, it's already a kind of blog 🙂 But, yes, I can do that.

It's not something I'd thought about before.

Though it's such a natural question in my eyes: Whoever exports data with EXPORT TO DATABASE should think about the time dependency of the involved data types!

Regards,

Rüdiger

[matt]

Oh absolutely. I've just never used EXPORT TO DATABASE.

[Former Member]

Hi Matthew,

yes, this is not my favorite approach neither, but as you mentioned further in this thread, there can be some issues in multi-server system with sync via memory (either using EXPORT or using shared objects). Then sometimes EXPORT TO DATABASE can do the job (if one does not want to sync via Z-table).

Just think of the scenario when you have to develop the user exit (which is part of update processing, i.e. after COMMIT WORK) where you need to react on some value set in the first part of the program (processed in DIA WP). As you cannot be sure whether UPD WP from the same appl. server as DIA WP will be selected, then you have to sync via database, not via memory.

Yes, I know - it should be rather passed as Z- update function module's parameter and then read within the user exit running in the UPD WP. But we already faced the scenario where one more COMMIT WORK in the middle caused splitting of update processing into two parts and in that case those two UPD FMs ran on different servers, IMHO (see my post about this: , so that using persistent data was the only option.

BR

Michal

[Former Member]

Hi, in my opinion - imagine how the data would get bigger if you save the metadata for every single record....

I would go from different side to the problem.

In your system there is everytime only one version of active VBAK dbtable&structure of course.

But in my solution I would probably keep every single version of VBAK since beginnig, eg. structures Z_VBAK_000, Z_VBAK_002, Z_VBAK_003. That should be structures only, no db tables, to know how the structure was in a specific time.

Next I would maintain a customization table maybe, where I would keep current version that is valid in the system, e.g. 'Z_VBAK_003'.

When exporting data to the database cluster, you can save also other data besides the cluster data itself, in custom columns. In one of the custom columns I would save this information - which data structure was valid in the time of saving.

The creating of versions Z_VBAK_00* and mainstaining of the current version could be automatized by relevant program I think...

So in db cluster I have data, and I know the ddict structure they correspond to.

Then when you need to read the data, you can check the strucure of the cluster (e.g. it was Z_VBAK_001), create data dynamically by this type/structure, import data to it, and then you need to have a custom conversion methods that will convert any historical structure to the current VBAK object or do just a move-corresponding...

Another possibility would be to serialize the structure VBAK to a xstring before saving CALL TRANSFORMATION, where an XML (compressed) will be saved instead of the structure inself.

Please take my writing as a abstract proposal and polemic only...

Br

Bohuslav

[Ruediger_Plantiko]

Hi, Bohuslav,

Hi, in my opinion - imagine how the data would get bigger if you save the metadata for every single record....

That's why I had written in the question:

(In an optimized version, the type info could be stored in a separate cluster, and being referenced by a version number only in the data cluster, for efficiency).

So your proposal is to save the historical DDIC types as own DDIC types.

One could even use the DDIC version from version management that was active when the EXPORT was performed: It would be a nice ABAP basis functionality to recover an old DDIC type in form of a type handle - the components have to be checked recursively for old versions (of include structures and component types), too. Don't know whether such a functionality exists.

Note (edited): Such a functionality would be of limited use, however, since in productive systems the version management usually is switched off.

For a general solution along your proposal, one would need lots of DDIC objects. Also, a DDIC type name which exploits the full allowed length would bring you in trouble with your proposed naming convention (but this could be resolved with a completely own name, and a name-mapping).

Not all exported types may be defined as DDIC structures. There are table types, type pools, types in classes (ZCL_TYPES=>TY_VBAK), local types in programs, and even dynamic types could be used. A general functionality should be independent of the storage type of the info.  This is why I would be interested in serializing CL_ABAP*DESCR instances: This would solve the problem in general.

Another possibility would be to serialize the structure VBAK to a xstring before saving CALL TRANSFORMATION, where an XML (compressed) will be saved instead of the structure inself.

An advantage of CALL TRANSFORMATION ID would be that it preserves the component names. But the type info itself is missing.

Please take my writing as a abstract proposal and polemic only...

Go ahead - I like polemics, as you see... 🙂 They teach us so much!

Kind regards,

Rüdiger

[matt]

wouldn't CALL TRANSFORMATION run into the same problem? If VBAK has changed since I stored the xstring in the database, I won't be able to convert it back.

Programmatically one solution would be to define local types within the method that deals with this - with a big comment "If you change this type information you won't be able to read old data!".

[Ruediger_Plantiko]

Yes, but compared to EXPORT TO DATABASE, CALL TRANSFORMATION at least preserves the component names. But the type infos are missing completely, so this is no alternative.

Programmatically one solution would be to define local types within the method that deals with this - with a big comment "If you change this type information you won't be able to read old data!".

Yes, generated code for the type information would be an alternative. But I am looking for something nearer to the RTTI. There are so many "by kernel module" methods and cool helper classes out there - maybe there is a serializer for RTTI objects available somewhere.

[Former Member]

The IMPORT statement works fine if target internal table has all fields of source internal table, plus some additional fields at the end, something like append structure of vbak.

Here is the snippet used.

1. TYPES:
2. BEGIN OF ty,
3.   a TYPE i,
4. END OF ty,
5. BEGIN OF ty2.
6.         INCLUDE TYPE ty.
7. TYPES:
8.   b TYPE i,
9. END OF ty2.
10. DATA: lt1 TYPE TABLE OF ty,
11.       ls TYPE ty,
12.       lt2 TYPE TABLE OF ty2.
13. ls-a = 2. APPEND ls TO lt1.
14. ls-a = 4. APPEND ls TO lt1.
15. EXPORT table = lt1 TO MEMORY ID 'ZTEST'.
16. IMPORT table = lt2 FROM MEMORY ID 'ZTEST'.

I guess IMPORT statement would behave fine if current VBAK has more fields than older VBAK.

/.

[Ruediger_Plantiko]

Manish,

thanks for this helpful remark! I checked your code on my system, and it works here, too.

Since the 99% case is that fields are appended, the IMPORT functionality, my example with VBAK and VBAP is not convincing any more: SAP will very rarely change existing database tables in other way  than appending fields at the end.

Regards,

Rüdiger

[Private_Member_7726]

Hi Rüdiger,

One exception seems to be FI-CA... Printing of certain FI-CA correspondence types that store "snapshots" of data in correspondence cluster DFKKCODCLUST gets broken during upgrades with such regularity (normaly because some of DFKKOP included structures gets extended) that there are series of OSS Notes like 1694243 - COTYP 0001: Printing after structural change, with no real solutions offered... The correspondence logic even foresees awkward versioning, but it involves copying DDIC types under new names and adding coding to convert from old types to new... so SAP don't do it (forget to do it) even themselves.

I fell into this trap by creating own correspondence types "SAP way", and aggravated it by storing full snapshot of necessary data on creation... now I have to copy close to 50 types of differing complexity when I need a new version. And what gets written to such clusters does not need to be DB table. Storing type information in such a form that data reference could be recreated from it would be nice and useful functionality.

cheers,

Janis

[Former Member]

2 problems

a) there are standard sap tables where customer has to do his appended fields "in the middle" to an append already prepared by sap; so not everytime customer append files are appended to the end of structure

b) the customer can later also change his append - and insert new fields before already existing

in both cases simple "unclustering" will fail i thing

lets clear the way how cluster and transform works, correct me if i am wrong

- cluster is just a bunch of bytes, something like a snapshot of memory; before saving to db is compressed by default; no metadata about what data belongs to which components are included, if you try to restore it to incorrect structure you get dump (in better case of incopatible values) or you will get ok result with shifted data segments (e.g. only character fields)

- transformation - at least some metadata are appended, data is transformed to an xml with overloaded size but it can be well compressed; structure component names are used to name the xml elements; if you try to unserialize, correct destination components are searched by the name; if found, value is passed to it, if not found value is not used anywhere, new components remain empty, so something like more-corresponging

in both cases you have not a full control about the process of unserialize/unclustering

I think to real stable & maximally generic solution it's really complicated, not easy but interesting theme and I suppose never will be solved by SAP, since clusters are not meant to be a long-therm data storage, and responsibility to structure changes are on developers

[Ruediger_Plantiko]

- cluster is just a bunch of bytes, something like a snapshot of memory; before saving to db is compressed by default; no metadata about what data belongs to which components are included,

Not completely true - the information on the elementary data types is contained in the cluster. You can recover a data object from a data buffer with method CL_ABAP_EXPIMP_UTILITIES=>DBUF_IMPORT_CREATE_DATA. The data object that is created by the method will contain all the correct elementary data types (but not the include substructures, not the data elements, not the currency reference information for currency amounts, ...)

and I suppose never will be solved by SAP

I am a bit more optimistic: CL_ABAP*DESCR can be used in ABAP syntax (CREATE DATA ... TYPE HANDLE LO_TYPE ). So maybe there is some support out there (or forthcoming) for serialization of those type info objects. And also the serialization of reference-free data structures into an XSTRING is far away from being obsolete. For example, in the SAP standard for Promotion Management for Retail (DMF), developed recently, it is heavily used by SAP developers.

[SuhaSaha]

Hello Rüdiger,

Thanks for tingling the braincells

The data are not lost, however: Using method CL_ABAP_EXPIMP_UTILITIES=>DBUF_IMPORT_CREATE_DATA, they can be recovered from an XSTRING.

I was wondering if this method can be used for data clusters as well

Not all exported types may be defined as DDIC structures. There are table types, type pools, types in classes (ZCL_TYPES=>TY_VBAK), local types in programs, and even dynamic types could be used.

IMHO the local types, dynamic types etc. are themselves not persistent. So this is why the standard EXPORT techniques may not support storing the metadata.

This is why I would be interested in serializing CL_ABAP*DESCR instances: This would solve the problem in general.

I would agree with you on this. If only we could serialize them

BR,

Suhas

[Ruediger_Plantiko]

I was wondering if this method can be used for data clusters as well 

If you work with 2 steps: export the data (TO DATA BUFFER) into an XSTRING, and in a second step, put this XSTRING on the database, you can get a cluster of ABAP data which can be reconstructed with the mentioned method.

If the data are already on the database (with no intermediate XSTRING export), say INDX, the question would be how to build the raw string from the CLUSTD field contents (maybe they have to be concatenated?) I don't know.

[Former Member]

Concatenation does not work.

I don't think we can analyze the bare cluster or data buffer with without having a format specification at hand.

Since both contents are hex starting with FF06, i converted the hex strings to ascii format using text editor.

For a single integer variable sent to data buffer, following things can be seen after conversion to ascii:

• 4103 - utf16le encoding
• formal parameter name
• variable value

For same variable sent to cluster, only 4103 was recognizable.

[Former Member]

Thanks Rudiger for posing this question. Kudos to all of you  who have tried to answer it with sound reasoning. After going through the discussion, I have two questions -

i) Issue primarily talked about  the danger when  export to data cluster command is used and before importing the data,  the structure is changed. Manish wrote a code to analyze the impact. He used export to memory id statement, while the discussion was about export to data cluster. Does export command store data in the same format internally irrespective of whether we are exporting to memory id or data cluster  or is there a difference between the two  . Were we trying to simplify things by applying boundary conditions.

Generally, we use data cluster for transient storage or to pass itab between two external sessions(between two tcodes ) and memory id  for passing itab  between the programs in a call sequence.

ABAP Keyword Documentation

ii) How did you test it.  I presume, you set a breakpoint on import to memory id statement. Then changed the structure and then pressed F5.  Then checked in the debugger, whether after import the new structure contained the added field. Many times, I have seen program dumping because of change of structure during runtime. ( Not sure, whether it dumps only for select statement or not ).

Any idea, when we activate a switch/business function set, where does the new append structure gets added. Is it always at the end or it can be anywhere in between.

[Ruediger_Plantiko]

Hi Debopriyo,

to resolve your doubts, I have modified Manish's program as follows:

form test.
   types:
   begin of ty1,
     a type i,
   end of ty1,
   begin of ty2,
     a type i,
     b type i,
   end of ty2.

   data: ls1 type ty1,
         lt1 type table of ty1,
         lt2 type table of ty2.

   if p_exp eq 'X'.
     ls1-a = 2. append ls1 to lt1.
     ls1-a = 4. append ls1 to lt1.
     export table = lt1 to database indx(qt) id 'ZTEST'.
   else.
     import table = lt2 from database indx(qt) id 'ZTEST'.
     break-point.
   endif.

endform.

I called it in export mode first (p_exp = 'X'), so the table was exported to database. Then I left the program with '/n', and started it again, now in import mode. At the breakpoint, table lt2 was filled with the expected values for column a and with initial values in column b.

The mechanism for exporting data objects is the same for all different layers for which it is provided: shared memory, shared buffer, database, or memory, or even a data buffer (xstring).

Using a cluster is not only, as you say, useful for passing "transient data" within runtime from one program part to another (for this case, there are better solutions in most situations). Instead, I saw the EXPORT TO ... DATABASE ... concept used for saving a cluster of data belonging to a "document" object, with the document ID as key and all the relevant parts belonging to that document contained in the data buffer.

Regards,

Rüdiger

[Former Member]

Thanks Rüdiger for the clarification. It was mea culpa that I didn't read every line of Manish's code ( missed out the data declaration part ). I presumed  data declaration referred to DB table ( instead of local type )  . So, was naturally curious how come the system didn't throw a dump when there is change of structure  during runtime.

You mentioned that there are better solutions for passing transient data within runtime from one program to another.  It would be helpful, if you can provide  me some solution for the scenario given below.

In my last project, one of my colleagues had to do pass itab between two badi methods of different BADI . The first method fills the itab, and then later on  in the call stack some subsequent  follow on documents are created. The second  badi method gets called during creation of follow on document.  Reading itab via import to memory id / storing in singleton class attribute/ global data of function module failed as the second program was running in another external session with its own context.  In that scenario, he used export/import  to data cluster.

Other ways of doing it  would be  storing/retrieving from a Z table, writing/ reading from a file , use shared objects.

[matt]

One issue to watch out for with shared objects is synchronisation between application servers. Perhaps it's not the case now, but it certainly used to be that you could put data in shared memory - but it's only shared by processes on the same app server.

[Ruediger_Plantiko]

Hi Debopriyo,

You mentioned that there are better solutions for passing transient data within runtime from one program to another.  It would be helpful, if you can provide  me some solution for the scenario given below.

I thought on self-developed solutions. In your example with BAdIs for existing standard processes, there is probably no alternative, since you have no influence on how the new internal session is created (the standard does this for you).

An example of a self-developed case: Did you know that arbitrary data types (except those containing references) are allowed for PARAMETERS of a report if they have the option NO-DISPLAY?

If you create the new internal session with SUBMIT AND RETURN, you can pass arbitrary complex data of the calling application - using WITH - to such a NO-DISPLAY parameter.

I have used this method successfully in some cases. These were cases were it was really necessary to have a fresh session for the report to work properly. If this is not necessary, it would be even better,  in such a case to provide an ENTRY subroutine and to call that subroutine externally, instead of SUBMITting the report. If the report makes only modest use of global data and offers proper ways for initializing them, this should work pretty well. (Best way, of course: The report is only a facade for a class, and all the work is done in a class. In this case, call the methods of that class directly.)

Regards,

Rüdiger