Raw segment data not grouping properly from the Feed file in my Custom Extension

[former_member771538]

In my custom extension (Quote) in the grouping phase, the raw segment(E1EDP01-ABFTZ) data is not being picked properly from Order05 IDOC.

Also, I don't see any errors logged. this is not happening in my local instance. Once I restart DataHub server with Create-Drop then Segment data is picked properly.

if data is loaded properly in the log I can see the segment data,see below log (Segment is delighted with Bold letters)

SAPORDER_INBOUND_POOL RawQuotes uuid=09fda963-0250-472b-be2c-6b3ee361eba6 id=97 delete=false action=10 fields={E1EDP01-MENEE=EA, E1EDK03-DATUM=20220624, E1EDP01-E1EDP19-KTEXT=UMV1422US OTR MICRO 1.4 CF 24IN SS, E1EDP01-MENGE=1.000, E1EDP01-E1EDP19-IDTNR=000000000UMV1422US, E1EDP01-NETWR=409.00, E1EDP01-CURCY=USD, E1EDS01-SUMME=4, E1EDP01-E1EDP05-KSCHL=null, EDI_DC40-MESTYP=ZORDERS, E1EDP01-E1EDP20-WMENG=1.000, E1EDKA1-PARVW=1B, E1EDK02-QUALF=001, E1EDP01-E1EDP19-QUALF=002, E1EDP01-POSEX=000010, E1EDK14-ORGID=01, E1EDK01-BSTZD=SING, E1EDP01-ABGRT=SO Double ordered by customer, E1EDK02-BELNR=7399001507, E1EDP01-ABGRU=SA, E1EDK14-QUALF=006, E1EDP01-ABFTZ=0.000, E1EDKA1-PARTN=0070210868, E1EDP01-E1EDP20-EDATU=20220628, E1EDP01-E1EDP05-KRATE=409.00, E1EDK01-CURCY=USD, E1EDP01-E1EDP05-KOTXT=Contractual INP, E1EDK01-BELNR=7399001507, E1EDK03-IDDAT=002, E1EDP01-WERKS=US41} traceId=54428c27-16b5-3724-bd0e-b23b9ff59d53 batchId=0000000924946877|R1NCLNT100|ORDERS05

If not picked properly then data is null for the segment, see below log (Segment is delighted with Bold letters)

SAPORDER_INBOUND_POOL RawQuotes uuid=b71cc9f6-a2d4-4c21-ae62-6307aecf2b82 id=134 delete=false action=13 fields={E1EDP01-MENEE=EA, E1EDK03-DATUM=20220624, E1EDP01-E1EDP19-KTEXT=UMV1422US OTR MICRO 1.4 CF 24IN SS, E1EDP01-MENGE=1.000, E1EDP01-E1EDP19-IDTNR=000000000UMV1422US, E1EDP01-NETWR=409.00, E1EDP01-CURCY=USD, E1EDS01-SUMME=4, E1EDP01-E1EDP05-KSCHL=null, EDI_DC40-MESTYP=ZORDERS, E1EDP01-E1EDP20-WMENG=1.000, E1EDKA1-PARVW=1B, E1EDK02-QUALF=001, E1EDP01-E1EDP19-QUALF=002, E1EDP01-POSEX=000010, E1EDK14-ORGID=01, E1EDK01-BSTZD=null, E1EDP01-ABGRT=null, E1EDK02-BELNR=7399001507, E1EDP01-ABGRU=SA, E1EDK14-QUALF=006, E1EDP01-ABFTZ=null, E1EDKA1-PARTN=0070210868, E1EDP01-E1EDP20-EDATU=20220628, E1EDP01-E1EDP05-KRATE=409.00, E1EDK01-CURCY=USD, E1EDP01-E1EDP05-KOTXT=Contractual INP, E1EDK01-BELNR=7399001507, E1EDK03-IDDAT=002, E1EDP01-WERKS=US41} traceId=895dde21-494c-3e53-8d2b-713030ad8ae1 batchId=0000000924946877|R1NCLNT100|ORDERS05

in both the cases same IDOC is sent and also remaining segment data is loading properly.

raw file data from the feed file

....

<E1EDP01 SEGMENT="1">
<POSEX>000010</POSEX>
<ACTION>001</ACTION>
<MENGE>1.000</MENGE>
<MENEE>EA</MENEE>
<PMENE>EA</PMENE>
<ABFTZ>0.000</ABFTZ>

<VPREI>409.00</VPREI>

........

........

</E1EDP01>

.....

facing similar issue for other the segment E1EDP01-ABGRT in the same extension

Can anyone help me in finding the cause to fix the issue? any idea why this inconsistent behavior?

Please let me know how raw segments get loaded and data picked in the grouping phase.

Data Hub version using : 6.7.0.5

Thank you.

[Slava]

Unfortunately I'm not familiar with the iDocs and sap integration packages, so I can't give a specific advice. But generally, if I remember correctly, DataHub groups items based on their type and key attributes. If you're seeing unexpected grouping results, then

1. check the item canonical type definitions in the extension.xml and specifically the key attributes (primaryKey is set to true). Raw items are grouped based on the key definitions of the corresponding canonical item, into which they will be composed. Note, a single raw item can be composed into a multiple canonical items of different type.
2. if you have custom grouping handlers, check the logic in them - perhaps they are not doing what you expect.

[former_member771538]

Thanks for your Reply,

Yes, I verified everything in the logs, and canonical types definitions and target mappings loaded properly.

But the issue is occurring before that item comes to the canonical phase, which means raw fragments not picking up properly while loading the feed (IDOC) data into RAW item(RawQuote item in this case).

[Slava]

What does it mean "raw fragments not picking up properly while loading the feed"? Does it mean some raw fragments are not loaded and do not exist as raw items? Or you mean all raw fragments are loaded and the raw items look correct, but when the canonical item is created some of the raw fragments are not applied to the canonical item?

The latter one is related to grouping. That could happen because the raw items are not grouped correctly. Each group of raw items represents a single canonical item that will be composed. That's why I'm suggesting to double check the mapping between the canonical items and raw items and keys of the canonical item. I'm not familiar with SAP domain and iDocs, as I've said already, so I will give an abstract example.

Lets say we have items of RawTypeA loaded:

attr1 | attr2 | attr3  | name
-----------------------------------
one1  | two1  | three1 | item one
one1  | two1  | three2 | item two
one1  | two2  | three1 | item three
one1  | two1  | three1 | item four

Now, let's say there are two canonical types: CanonicalTypeA and CanonicalTypeB, which are composed from the RawTypeA. CanonicalTypeA has two primary key attributes, which are mapped to the attr1 and attr2 attributes in the raw type; and CanonicalTypeB has three primary key attributes, which are mapped to attr1, attr2 and attr3 respectively.

With such setup during composition the raw items will be grouped based on the canonical type and key attributes as follows.
For CanonicalTypeA there will be two raw item groups, which will create two canonical items:

1. item one, item two, item four
2. item three

For CanonicalTypeB there will be three raw items groups, which will create three canonical items:

1. item one, item four
2. item two
3. item three

As you can see the grouping is affected by how Canonical Item is mapped to raw items, and that's the reason I'm asking you to check the metadata.

If, let's say, there are items of RawTypeB loaded too, but they are not appearing in the expected groups, probably that is because the canonical type is not mapped to RawTypeB at all, or the raw type B attributes corresponding to key attributes in the canonical item have different/unexpected values.

[former_member771538]

What does it mean "raw fragments not picking up properly while loading the feed"? Does it mean some raw fragments are not loaded and do not exist as raw items?

YES. issue in mapping IDOC file to the Raw fragments (My bad, I should have conveyed it in a better manner)

I have an XML feed file as an example with segments below(this is some part of the complete file)

....

.....

<E1EDP01 SEGMENT="1">
<POSEX>000010</POSEX>
<ACTION>001</ACTION>
<MENGE>1.000</MENGE>
<MENEE>EA</MENEE>
<PMENE>EA</PMENE>
<ABFTZ>0.000</ABFTZ>

<VPREI>409.00</VPREI>

....

....

In my custom extension Quote which don't have any OOTB dependencies, I have below raw fragments defilned in raw file

<type>RawQuotes</type>
<description>Order Data coming SAP ERP (IDoc Type ORDERS05)
</description>
<attributes>
<attribute>
<name>E1EDK01-BELNR</name>
</attribute>
<attribute>
<name>E1EDP01-POSEX</name>
</attribute>
<attribute>
<name>E1EDP01-ABFTZ</name>
</attribute>
<attribute>
<name>E1EDP01-ABGRU</name>
</attribute>
<attribute>
<name>E1EDP01-E1EDP20-WMENG</name>
</attribute>
<attribute>
<name>E1EDP01-E1EDP20-EDATU</name>
</attribute>
<attribute>
<name>EDI_DC40-MESTYP</name>

</attribute>

.....

.....

</attributes>

</item>

my expectation is after proper mapping of the XML data to my Raw fragments, I should see the data for my raw fragment E1EDP01-ABFTZ=0.000

this mapping is very inconsistent, some times I am able to see proper raw fragment data mapped and sometimes null (means not the data in the feed file), this inconsistency is happening after the Datahub server restart.

if data is loaded in the raw fragment properly after a restart, i is working for all the XML feeds, otherwise mapping resulting null value to raw fragment data.

if data is loaded properly in the log I can see the segment data,see below log (Segment is delighted with Bold letters)

SAPORDER_INBOUND_POOL RawQuotes uuid=09fda963-0250-472b-be2c-6b3ee361eba6 id=97 delete=false action=10 fields={E1EDP01-MENEE=EA, E1EDK03-DATUM=20220624, E1EDP01-E1EDP19-KTEXT=UMV1422US OTR MICRO 1.4 CF 24IN SS, E1EDP01-MENGE=1.000, E1EDP01-E1EDP19-IDTNR=000000000UMV1422US, E1EDP01-NETWR=409.00, E1EDP01-CURCY=USD, E1EDS01-SUMME=4, E1EDP01-E1EDP05-KSCHL=null, EDI_DC40-MESTYP=ZORDERS, E1EDP01-E1EDP20-WMENG=1.000, E1EDKA1-PARVW=1B, E1EDK02-QUALF=001, E1EDP01-E1EDP19-QUALF=002, E1EDP01-POSEX=000010, E1EDK14-ORGID=01, E1EDK01-BSTZD=SING, E1EDP01-ABGRT=SO Double ordered by customer, E1EDK02-BELNR=7399001507, E1EDP01-ABGRU=SA, E1EDK14-QUALF=006, E1EDP01-ABFTZ=0.000, E1EDKA1-PARTN=0070210868, E1EDP01-E1EDP20-EDATU=20220628, E1EDP01-E1EDP05-KRATE=409.00, E1EDK01-CURCY=USD, E1EDP01-E1EDP05-KOTXT=Contractual INP, E1EDK01-BELNR=7399001507, E1EDK03-IDDAT=002, E1EDP01-WERKS=US41} traceId=54428c27-16b5-3724-bd0e-b23b9ff59d53 batchId=0000000924946877|R1NCLNT100|ORDERS05

If not picked properly then data is null for the segment, see below log (Segment is delighted with Bold letters)

SAPORDER_INBOUND_POOL RawQuotes uuid=b71cc9f6-a2d4-4c21-ae62-6307aecf2b82 id=134 delete=false action=13 fields={E1EDP01-MENEE=EA, E1EDK03-DATUM=20220624, E1EDP01-E1EDP19-KTEXT=UMV1422US OTR MICRO 1.4 CF 24IN SS, E1EDP01-MENGE=1.000, E1EDP01-E1EDP19-IDTNR=000000000UMV1422US, E1EDP01-NETWR=409.00, E1EDP01-CURCY=USD, E1EDS01-SUMME=4, E1EDP01-E1EDP05-KSCHL=null, EDI_DC40-MESTYP=ZORDERS, E1EDP01-E1EDP20-WMENG=1.000, E1EDKA1-PARVW=1B, E1EDK02-QUALF=001, E1EDP01-E1EDP19-QUALF=002, E1EDP01-POSEX=000010, E1EDK14-ORGID=01, E1EDK01-BSTZD=null, E1EDP01-ABGRT=null, E1EDK02-BELNR=7399001507, E1EDP01-ABGRU=SA, E1EDK14-QUALF=006, E1EDP01-ABFTZ=null, E1EDKA1-PARTN=0070210868, E1EDP01-E1EDP20-EDATU=20220628, E1EDP01-E1EDP05-KRATE=409.00, E1EDK01-CURCY=USD, E1EDP01-E1EDP05-KOTXT=Contractual INP, E1EDK01-BELNR=7399001507, E1EDK03-IDDAT=002, E1EDP01-WERKS=US41} traceId=895dde21-494c-3e53-8d2b-713030ad8ae1 batchId=0000000924946877|R1NCLNT100|ORDERS05

As the mapping itself has issues so grouping will not be as desired and so, Canonical data will not be accurate.

Any idea what could be the reason