Preventing Data Loss During Replication with Partial Data Model

[Beykan]

Introduction

This article discusses a strategy for preventing the unintended deletion of reduced XML payloads.

When reduced XML payloads are received from third-party systems like PTC, PDM, Salesforce, etc. SAP's standard behavior is to replace the existing target data objects with the incoming payload content. This means that any fields not included in the reduced payload could be unintentionally deleted in the target data object. This is because SAP interprets the absence of certain fields in the incoming data as an instruction to clear those fields in the existing data object.

Remark: The presence of middleware, such as SAP PI/PO, Seeburger, Jitterbit, etc., between third-party systems and SAP systems, plays a critical role in managing system integration; especially when dealing with different data model structures between the sender and receiving systems.

To mitigate this risk, it's crucial to implement certain strategies. Our strategy here in this particular case is pre-processing. Before processing the incoming XML payload, perform a pre-check to compare it with the currently active data object and decide which fields should be updated and which ones should be preserved.

For your purpose, where product data is being replicated using SAP's standard service provider ProductMDMBulkReplicateRequest_In, handling reduced XML payloads becomes even more critical. This service is designed to handle bulk replication of product data, and it’s essential to ensure that the received payloads are complete enough to maintain data integrity in SAP.

Background

When replication involves these third-party systems, the reduced or altered data payloads can lead to unintended consequences, such as missing or incomplete updates in SAP. This is especially critical when the third-party system's data model does not fully align with SAP’s, or when not all data from the third-party system is available or transmitted during replication.

Some key points to consider when integrating a third-party system with SAP:

1. Data Model Alignment
2. Handling Incomplete Data
3. Data Availability
4. Reduced payload approach: can all fields not required be reduced?
5. Etc.

Remark: When third-party systems send reduced XML payloads, which might omit certain fields, SAP may interpret these omissions as an instruction to delete or clear those fields. To prevent unintended data loss, it’s important to develop strategies to recognize and handle these cases, ensuring that only the intended updates are made.

Here is an example of a reduced XML payload that might be received from a third-party system:

Figure 1 – An example of reduced XML payload received from a third-party system

More detailed information about the ProductMDMBulkReplicateRequest_In service and the corresponding product data structure in SAP S/4HANA can be found on the official SAP Help Portal. 

Problem

When a third-party payload omits certain fields that are already populated in SAP, the SAP system may interpret these missing fields as blank or null during the inbound processing. This can result in previously enriched fields in SAP being overwritten and set to empty if they aren't included in the incoming payload. This behavior underscores the importance of ensuring that critical fields are always included in the payload or implementing logic to preserve existing data in SAP when certain fields are absent.

Feasible Option - Abstract

To address the issue of missing fields in third-party payloads leading to overwriting of populated fields in SAP, a Business Add-In (BAdI) can be implemented during inbound processing. Our applied custom logic in this BAdI ensures that fields already populated in SAP are not overwritten with null values if the corresponding fields are absent in the third-party payload. This solution helps maintain data integrity by preserving existing values in SAP when certain fields are not provided in the incoming data.

This article presents a possible solution to address missing data in the incoming payload by using the BAdI MDM_SE_PRD_BULK_REP_REQ. The idea is to detect differences between the active data in SAP and the incoming payload, ensuring that any field missing in the reduced payload is properly identified and managed.

Some Nice-To-Have Skills

Here are some nice-to-have skills and knowledge areas for handling this topic:

ABAP programming, T-code: SRT_MONI, Web Services, End-to-End Data Process Flow, XML/XSD, SAP Integration.

Solution in a Nutshell

Our idea is to dynamically identify all active fields within the Material Data Model in SAP during inbound processing that are not managed by the third-party system but are populated in the SAP system. We use BAdI MDM_SE_PRD_BULK_REP_REQ for this purpose.

Steps to Realize the Solution

For our solution, we pursued the following steps:

1. Create enhancement implementation for BAdI MDM_SE_PRD_BULK_REP_REQ.
2. To fill & populate the basic data segment of the XML payload, we created the method FILL_FROM_MARA.
3. To identify fields in the Material Master (MARA) that are not empty, we used the method cl_abap_struct_utilities=>filled_components. This ABAP class method allows you to determine which components of a structure (in this case, MARA) are populated. By leveraging this, you can focus on fields that are already populated in SAP and ensure that they are preserved and not unintentionally overwritten during data replication or integration processes with third-party systems.
4. Please see some code samples from method FILL_FROM_MARA. The import parameter is IV_MATNR and there is a changing parameter CS_OUT that holds the XML payload coming from the framework. This takes place in IF_MDM_SE_PRD_BULK_REP_REQ~INBOUND_PROCESSING:

Signature of IF_MDM_SE_PRD_BULK_REP_REQ~INBOUND_PROCESSING:

Figure 2 – Signature of method: IF_MDM_SE_PRD_BULK_REP_REQ~INBOUND_PROCESSING

The code for method IF_MDM_SE_PRD_BULK_REP_REQ~INBOUND_PROCESSING is like this:

METHOD if_mdm_se_prd_bulk_rep_req~inbound_processing.

DATA lv_matnr TYPE matnr.

lv_matnr = in-product-receiver_product_internal_id.

me->fill_from_mara(
  EXPORTING
    iv_matnr = lv_matnr " Material Number
  CHANGING
    cs_out   = out ).   " Product Replication Services Structure

COMMIT WORK.

ENDMETHOD.

Signature of FILL_FROM_MARA:

Figure 3 – Signature of method: FILL_FROM_MARA

The code for method FILL_FROM_MARA is like this:

METHOD fill_from_mara.

    FIELD-SYMBOLS:
      <lv_out>    TYPE any,
      <lv_active> TYPE any,
      <lv_out_x>  TYPE any.
    
    " select the active MARA data
    SELECT SINGLE * FROM mara
      INTO (ls_mara)
      WHERE matnr = _matnr    .      

    " check if it is a change scenario, then process
    IF sy-subrc NE 0.
      RETURN.
    ENDIF.

    " get the fields which are not empty from MARA result str
    cl_abap_struct_utilities=>filled_components(
      EXPORTING
        struct       = ls_mara
      RECEIVING
        filled_comps = DATA(lt_filled_comps) ).

    " assign the related OUT parameters to field symbols
    READ TABLE cs_out-product-mara_tab ASSIGNING FIELD SYMBOL(<ls_mara_out>) INDEX 1.
    READ TABLE cs_out-product-mara_x_tab ASSIGNING FIELD-SYMBOL(<ls_mara_x_out>) INDEX 1.

    " if related field symbols are not assigned then exit the method
    IF <ls_mara_out> IS ASSIGNED AND
       <ls_mara_x_out> IS ASSIGNED.

      " loop at filled info of MARA fields
      LOOP AT lt_filled_comps ASSIGNING FIELD-SYMBOL(<ls_filled_comps>).

        UNASSIGN <lv_out>.
        UNASSIGN <lv_active>.
        UNASSIGN <lv_out_x>.

       " assign the field not empty in MARA, for OUT parameter
       ASSIGN COMPONENT <ls_filled_comps>-name OF STRUCTURE <ls_mara_out> TO <lv_out>.

       " if related out field is already have a value, continue. Otherwise take an action
       IF <lv_out> IS ASSIGNED AND <lv_out> IS INITIAL.

         " assign the related fields value of MARA result
         ASSIGN COMPONENT <ls_filled_comps>-name OF STRUCTURE ls_mara TO <lv_active>.

         " if the field has a value in MARA (it has a value because of the loop logic), give it to OUT parameter
         IF <lv_active> IS ASSIGNED AND
            <lv_active> IS NOT INITIAL.

           " need to set update flag
           ASSIGN COMPONENT <ls_filled_comps>-name OF STRUCTURE <ls_mara_x_out> TO <lv_out_x>.

           " only update values which has update flag
           IF <lv_out_x> IS ASSIGNED.
              <lv_out> = <lv_active>.
              <lv_out_x> = abap_true.
            ENDIF.
          ENDIF.
        ENDIF.
      ENDLOOP.
    ENDIF.
  ENDMETHOD.

Conclusion

Integrating third-party applications with SAP can lead to challenges when the payloads are reduced, causing SAP to treat missing fields as blanks or nulls. To address this, we've implemented an inbound BAdI solution that detects and populates missing fields within the XML payload. This approach helps solving the issue of the deletion of populated fields in SAP.