SAP Marketing Cloud – Connect any email service pr...

mathias_knura · ‎2018 Jun 25

This blog post of how to connect any email service provider to SAP Marketing Cloud, consists of 3 chapter:

Email Send Scenario

Example ESP: SAP Digital Interconnect

Get Bounces

Get Bounces

This chapter explains how bounces can be pulled from email service provider along with an example implemantation for SAP Digital Interconnect.

Interface definitions

To get a general idea about what we try to accomplish, first a short overview of the bounce interface of the generic ESP adapter and the corresponding API of SAP Digital Interconnect will be given in this section. A full interface definition can be found in Integration Guide SAP Marketing Cloud Generic Email Integration.

SAP Marketing Cloud generic ESP adapter

GET Request

The generic ESP adapter of SAP Marketing Cloud sends a HTTP GET request to the endpoint /bounces with the following URL parameters:

startTimeUTC:
Describes the start time of a time frame in which the bounces to retrieve have occurred.

endTimeUTC:
Describes the end time of a time frame in which the bounces to retrieve have occurred.

page:
For very long result lists, most ESPs do not send back all results in a single response but in multiple pages of about 100 results. This parameter determines which page of the result list should be retrieved. The first page is always queried by sending a page value of 0.

sourceSystem:
Identifies the system which send the HTTP request.

After all, a complete GET request send by Marketing Cloud could look like this:

<HCI Tenant URL>/bounces?startTimeUTC=20171014132922&endTimeUTC=20171024142000&page=0&sourceSystem=TestSystem1

Expected response

The response to the request defined above should have the following structure:

HTTP Response from HCI:

{

    "page": "0",

    "lastPage": "true",

    "bounces": [

        {

            "messageId": "100000328097215",

            "recipient": "test1@example.com",

            "errorCode": "5.4.7",

            "errorText": "[internal] exceeded max time without delivery",

            "type": "Soft",

            "timestamp": "2017-10-22 11:44:18.779"

        },

        {

            "messageId": "100000328097294",

            "recipient": "test2@example.com",

            "errorCode": "5.4.7",

            "errorText": "[internal] exceeded max time without delivery",

            "type": "Soft",

            "timestamp": "2017-10-22 11:43:54.793"

        },

        {

            "messageId": "100000329705936",

            "recipient": "test3@example.com",

            "errorCode": "550",

            "errorText": "NOMB - 550 Requested action not taken: mailbox unavailable",

            "type": "Hard",

            "timestamp": "2017-10-24 14:02:06.055"

        }

    ]

}

While the property 'page' serves as identifier of the current package of bounces, the property 'lastPage' indicates whether there are some more packages that can be retrieved by sending further requests. Thus, the last package for the given timeframe provides the value true for property 'lastPage'

SAP Digital Interconnect

As most ESPs, SAP Digital Interconnect provides its own individual API for communication via HTTP requests which will be described in the following section. However, this section only considers the endpoints and properties relevant for the presented scenario. A complete overview of SAP Digital Interconnect's HTTP API can be found here.

GET Request to SAP Digital Interconnect

SAP Digital Interconnect's HTTP API for retrieving bounces is being contacted by sending a HTTP GET request to the endpoint /bounce with the followoing URL parameters:

startUTCTime:
Describes the start time of a time frame in which the bounces to retrieve have occurred.

endUTCTime:
Describes the end time of a time frame in which the bounces to retrieve have occurred.

groupIndex:
Digital Interconnect describes its result packages as 'groups', thus the 'groupIndex' property determines which result packages should be retrieved like SAP Marketing Cloud's page property. In contrast to SAP Marketing Cloud's implementation, the first page/group is always queried by sending a page value of 1.

campaign:
This property is a user defined identifier for a group of notifications.

Sample request:

<SAP Digital Interconnect API URL>/bounce?startUTCTime=20171014132922&endUTCTime=20171024140300&groupIndex=1&campaign=ABDCLNT100

Response from SAP Digital Interconnect

Digital Interconnect's responses have the following structure:

HTTP Response from SAP DI:

{

    "SAPnotificationList": {

        "groupIndex": 1,

        "groupCount": 1,

        "SAPnotifications": [

            {

                "account": "TestAccount",

                "notificationId": "100000328097215",

                "status": [

                    {

                        "statusCode": "CH_REJECTED",

                        "statusText": "SOFT_BOUNCE",

                        "channel": "EMAIL",

                        "timestamp": "2017-10-22 11:44:18.779",

                        "recipient": "email:test1@example.com",

                        "channelStatusCode": "SMTP=554,DSN=5.4.7",

                        "channelStatusText": "ERR - 554 5.4.7 [internal] exceeded max time without delivery"

                    }

                ],

                "characterSet": null,

                "sender": "sender1@exampe.com",

                "recipients": null,

                "contentTextType": null,

                "contentText": null,

                "contentTextEncoding": null,

                "channelPreferences": null,

                "configuration": null,

                "contentAttachments": null,

                "campaign": "TestCampaign"

            },

            {

                "account": "TestAccount",

                "notificationId": "100000328097294",

                "status": [

                    {

                        "statusCode": "CH_REJECTED",

                        "statusText": "SOFT_BOUNCE",

                        "channel": "EMAIL",

                        "timestamp": "2017-10-22 11:43:54.793",

                        "recipient": "email:test2@example.com",

                        "channelStatusCode": "SMTP=554,DSN=5.4.7",

                        "channelStatusText": "ERR - 554 5.4.7 [internal] exceeded max time without delivery"

                    }

                ],

                "characterSet": null,

                "sender": "sender1@exampe.com",

                "recipients": null,

                "contentTextType": null,

                "contentText": null,

                "contentTextEncoding": null,

                "channelPreferences": null,

                "configuration": null,

                "contentAttachments": null,

                "campaign": "TestCampaign"

            },

            {

                "account": "TestAccount",

                "notificationId": "100000329705936",

                "status": [

                    {

                        "statusCode": "CH_REJECTED",

                        "statusText": "HARD_BOUNCE",

                        "channel": "EMAIL",

                        "timestamp": "2017-10-24 14:02:06.055",

                        "recipient": "email:test3@example.com",

                        "channelStatusCode": "SMTP=550,DSN=null",

                        "channelStatusText": "NOMB - 550 Requested action not taken: mailbox unavailable"

                    }

                ],

                "characterSet": null,

                "sender": "sender1@exampe.com",

                "recipients": null,

                "contentTextType": null,

                "contentText": null,

                "contentTextEncoding": null,

                "channelPreferences": null,

                "configuration": null,

                "contentAttachments": null,

                "campaign": "TestCampaign"

            }

        ]

    }

}

The property 'groupCount' in Digital Interconnect 's responses mark the total number of packages for the requested time frame. Therefore, the last package is reached as soon as the values of 'groupIndex' and 'groupCount' are identical.

HCI implementation

General structure of the iFlow

The picture below shows the structure of the HCI iFlow used to enable the communication between SAP Marketing Cloud and SAP Digital Interconnect for the collecting bounces scenario. In the following sections, the different transformation steps within the flow are discussed in detail.

Configuration of Inbound- and Outbound connections

Defining the HCI endpoint

The endpoint URL of a generic iFlow on HCI is defined by:
<HCI Tenant iFlow URL>/<used protocol>/<defined endpoint>

While the <HCI Tenant iFlow URL> is always fixed for a HCI Tenant, the <used protocol> and <defined endpoint> can be configured directly within the iFlow. To perform this configuration, the connector starting from Sender must be double clicked:

In the next window, the Adapter Type can be defined on the right side of the General Tab, while the endpoint settings are made in the Adapter Specific Tab under Request Processing. The configuration for our bounce scenario is shown below:

With this configuration, the complete endpoint URL depending on the used HCI Tenant reads as: <HCI Tenant iFlow URL>/http/hybris_esp/bounces. After saving and deploying the configuration, a GET request to the specified URL should return with status 200.

Establishing connection to SAP Digital Interconnect

Similar as above, the connection settings for outgoing calls from HCI to any endpoint can be defined by double clicking the connector leading to Receiver node:

Here the connection settings can be maintained in the same way as for inbound connections within the tabs General and Adapter Specific. The configuration used in our bounce scenario is shown below:

In the field Address, the respective endpoint of SAP Digital Interconnect for collecting bounces has to be entered. If the endpoint requires some kind of authentication, the respective settings can be defined in a seperate authentication object which has to be maintained in field Credential Name. In case of the presented scenario, an authentication object with the user name and passwort of an internal SAP Digital Interconnect account has been created. More information regarding authentication objects can be found Authenticating from SAP Cloud Platform Integration.

.

Redefinition of the input parameters

The first task of HCI in our bounce scenario is to manipulate the structure of the GET Request send by SAP Hybris Marketing Cloud to fit SAP Digital Interconnect's API. To achieve this, the four URL parameters of the initial GET request have to be changed in an appropriate way. A glance at section 1.1.1 and 1.2.1 of this guide shows that the following parameter changes are necessary:

startTimeUTC → startUTCTime

endTimeUTC → endUTCTime

sourceSystem → campaign

page → groupIndex, starting with 1 instead of 0

This job is done by a script node called "Prepare Request" which has been inserted between the start event of our iFlow and the "Request-Reply" event that send the GET Request to SAP Digital Interconnect.

Since the URL parameters are stored in the header of the incoming GET request as one coherent string, the only way to modify them is using basic string operations:

Prepare Request Script:

def Message redefineInputHeaders(Message message) {   



    def header = message.getHeaders();

    def query = header.CamelHttpQuery;



    query = query.replaceAll('startTimeUTC','startUTCTime');

    query = query.replaceAll('endTimeUTC','endUTCTime');

    query = query.replaceAll('page','groupIndex');

    query = query.replaceAll('sourceSystem','campaign');



    regexPattern = "(?<=groupIndex=)[0-9]([0-9])?" as String;

    pattern = Pattern.compile(regexPattern) as Pattern;

    def matcher = pattern.matcher(query) as Matcher;

    def newQuery = "";



    if(matcher.find()){

        def page = query.substring(matcher.start(), matcher.end()) as Integer;

        page = page+1;

        newQuery = query.substring(0,matcher.start())+page+query.substring(matcher.end());

    }



    header.CamelHttpQuery = newQuery;

    message.setHeaders(header);



    return message;

}

In order to retrieve the digits that represent the page/groupIndex number within the parameter string, a regular expression is used. This expression stores the one or possibly two digits following in the substring groupIndex into a variable which is then increased by 1. When all parameters have been modified, the new query is stored in the header of the GET Request. After this, the modified message is sent to SAP Digital Interconnect.

Handling of the response from SAP Digital Interconnect

As soon as the response from SAP Digital Interconnect is given back to HCI, subsequent modification steps can be done. First of all, the reponse body is converted from JSON to XML. This step is necessary, because most of the message transformation nodes of HCI only work for XML content.

At the end of the iFlow, before returning the message to SAP Marketing Cloud, a reverse conversion to JSON will be applied. In the meantime, the following steps are performed.

Property mapping

Message mapping nodes facilitate an easy mapping of property values from a source element to a target element, representing the ingoing and outgoing flow. In order to perform a message mapping, the structures of the source and target element must be defined in seperate XSD-files that are given as input to the node:

XSD: Structure of SAP DI response:

<xs:schema attributeFormDefault="unqualified" elementFormDefault="qualified" xmlns:xs="http://www.w3.org/2001/XMLSchema">

  <xs:element name="SAPnotificationList">

    <xs:complexType>

      <xs:sequence>

        <xs:element type="xs:byte" name="groupIndex"/>

        <xs:element type="xs:byte" name="groupCount"/>

        <xs:element name="SAPnotifications" maxOccurs="unbounded">

          <xs:complexType>

            <xs:sequence>

              <xs:element type="xs:string" name="account"/>

              <xs:element type="xs:long" name="notificationId"/>

              <xs:element name="status" maxOccurs="unbounded" minOccurs="0">

                <xs:complexType>

                  <xs:sequence>

                    <xs:element type="xs:string" name="statusCode"/>

                    <xs:element type="xs:string" name="statusText"/>

                    <xs:element type="xs:string" name="channel"/>

                    <xs:element type="xs:string" name="timestamp"/>

                    <xs:element type="xs:string" name="recipient"/>

                    <xs:element type="xs:string" name="channelStatusCode"/>

                    <xs:element type="xs:string" name="channelStatusText"/>

                  </xs:sequence>

                </xs:complexType>

              </xs:element>

              <xs:element type="xs:string" name="characterSet"/>

              <xs:element type="xs:string" name="sender"/>

              <xs:element type="xs:string" name="recipients"/>

              <xs:element type="xs:string" name="contentTextType"/>

              <xs:element type="xs:string" name="contentText"/>

              <xs:element type="xs:string" name="contentTextEncoding"/>

              <xs:element type="xs:string" name="channelPreferences"/>

              <xs:element type="xs:string" name="configuration"/>

              <xs:element type="xs:string" name="contentAttachments"/>

              <xs:element type="xs:string" name="campaign"/>

            </xs:sequence>

          </xs:complexType>

        </xs:element>

      </xs:sequence>

    </xs:complexType>

  </xs:element>

</xs:schema>

XSD: Structure of response to SAP Marketing Cloud:

<xs:schema attributeFormDefault="unqualified" elementFormDefault="qualified" xmlns:xs="http://www.w3.org/2001/XMLSchema">

  <xs:element name="root">

    <xs:complexType>

      <xs:sequence>

        <xs:element type="xs:byte" name="page"/>

        <xs:element type="xs:string" name="lastPage"/>

        <xs:element name="bounces" maxOccurs="unbounded" minOccurs="0">

            <xs:complexType>

                <xs:sequence>

                    <xs:element type="xs:long" name="messageId"/>

                    <xs:element type="xs:string" name="recipient"/>

                    <xs:element type="xs:string" name="errorCode"/>

                    <xs:element type="xs:string" name="errorText"/>

                    <xs:element type="xs:string" name="type"/>

                    <xs:element type="xs:string" name="timestamp"/>

               </xs:sequence>

            </xs:complexType>

        </xs:element>

      </xs:sequence>

    </xs:complexType>

  </xs:element>

</xs:schema>

With these two structures used as input for the message mapping node, the mapping in our bounce scenario looks as follows:

While performing the property mapping, it is also possible to assign additional logic like arithmetical operations or string manipulations by clicking on a connector. In the bounce scenario, this has been done for mapping pairs groupIndex/page and groupCount/lastPage. Since the page count logic of SAP Digital Interconnect is different than the one implemented in SAP Marketing Cloud, we substract 1 from groupCount while mapping it to property page. At the same time, the values for source properties groupIndex and groupCount are compared in order to determine if the last page is reached. If yes, the target property lastPage is set to true, otherwise to false.

Handling of empty result set

Before routing the message to the previously presented message mapping node, it has to be checked if there are any values in the result set bounces coming back from SAP DI. The reason for this is a known bug that causes the HCI to crash if an empty array is given to the mapping engine.

This is done by using a router which checks whether the number of results is equal to 0. If this is the case, the message is routed to a branch that skips the mapping and sends a default response back to SAP Marketing Cloud:

Default response for 0 result:

{

    "page": 0,

    "lastPage": true,

    "bounces": []

}

Cosmetics: retrieve error code and bounce type

SAP Marketing Cloud not only has certain requirements regarding the structure of the incomig reponses, but also needs the entities to provide standardized values that can be easily processed. That's why the response value for entities errorCode, errorText and type have to be adjusted by HCI in order to fit these requirements. As seen in section above, the values returned by SAP Digital Interconnect need to be transformed in the following way:

Cange error code/text:

def Message changeErrorCode(Message message){



    def body = message.getBody(java.lang.String) as String;

    def root = new XmlSlurper().parseText(body);



    root.bounces.each { bounce ->

        def errorCode = bounce.errorCode.text();

        def errorText = bounce.errorText.text();



        def regexPattern = "\\[(internal)]" as String;

        def pattern = Pattern.compile(regexPattern) as Pattern;

        def matcher = pattern.matcher(errorText) as Matcher;



        if(matcher.find()){

            errorText = errorText.substring(matcher.start());

            bounce.errorText.replaceBody(errorText);

        }



        regexPattern = "[0-9][.][0-9][.][0-9]" as String;

        pattern = Pattern.compile(regexPattern) as Pattern;

        matcher = pattern.matcher(errorCode) as Matcher;



        if(matcher.find()){

            def code = errorCode.substring(matcher.start(),matcher.start()+5);

            bounce.errorCode.replaceBody(code);

        } else {

            regexPattern = "(DSN)=[0-9][0-9][0-9]" as String;

            pattern = Pattern.compile(regexPattern) as Pattern;

            matcher = pattern.matcher(errorCode) as Matcher;



            if(matcher.find()){

                def code = errorCode.substring(matcher.end()-3,matcher.end());

                bounce.errorCode.replaceBody(code);

            } else {

                regexPattern = "(SMTP)=[0-9][0-9][0-9]" as String;

                pattern = Pattern.compile(regexPattern) as Pattern;

                matcher = pattern.matcher(errorCode) as Matcher;



                if(matcher.find()){

                    def code = errorCode.substring(matcher.end()-3,matcher.end());

                    bounce.errorCode.replaceBody(code);

                }

            }

        }

    }



    message.setBody(XmlUtil.serialize(root));



    return message;

}

Change bounce type:

def Message changeType(Message message){



    def body = message.getBody(java.lang.String) as String;

    def root = new XmlSlurper().parseText(body);



    root.bounces.each { bounce ->

        def type = bounce.type.text();



        if (type == "SOFT_BOUNCE") {

            bounce.type.replaceBody("Soft");

        }   



        if (type == "HARD_BOUNCE") {

            bounce.type.replaceBody("Hard");

        }

    }



    message.setBody(XmlUtil.serialize(root));



    return message

}

Single result handling

By reason of another error in the HCI's XML to JSON Converter we are not finished yet, but have to apply one more step to our iFlow. In case there is exactly one bounce returned by SAP Digital Interconnect, the XML to JSON Converter does not create an array of objects with one element as value for the result entity bounces, but omits the array and only creates a plain bounce object. This response causes the ABAP XML parser of SAP Marketing Cloud to crash and therefore needs to be corrected. This job is done by the following method:

Handle single bounces:

def Message handleSingleBounce(Message message){

    def slurper = new JsonSlurper();

    def body = message.getBody(java.lang.String) as String;



    def bodyJSON = slurper.parseText(body); 

    def bounces = bodyJSON.bounces;

    def bouncesAsString = JsonOutput.toJson(bounces);



    if (!bouncesAsString.startsWith("[")){

        def bouncesWithBrackets = slurper.parseText("["+bouncesAsString+"]");

        bodyJSON.bounces = bouncesWithBrackets;



        message.setBody(JsonOutput.toJson(bodyJSON));

    }



    return message;

}

The method simply checks whether the value for entity bounces starts with a bracket "[". If this is not the case, the whole statement is enclosed in brackets and given back.

Previous Chapter:

Email Send Scenario

Example ESP: SAP Digital Interconnect