Deep Dive 16 with SAP Cloud SDK - An End to End story of the integration of an OData service with SAP Conversational AI (also known as Recast AI)

[former_member588271]

The objective of this deep dive is to teach how to link a chatbot created on SAP Conversational AI (also known as Recast.AI) with an ERP system via OData services. After this deep dive you should be able to extent your bot to OData services. This deep dive is part of a series, which you can check out SAP Cloud SDK.

Note:

• This tutorial requires access to:
  
  
  
  • An SAP ERP system – SAP S/4HANA preferred, but any system with RFC or OData API works
  
  
  • An SAP Conversational AI account (can also be created for free)
  
  
  
  


• This post is part of a series and builds on some previous tutorial

Goal of this post

After following this deep dive, you will be able to enrich your chatbots with real time data straight from your SAP S/4HANA. Further, the chat bot will be deployable on many different channels like Facebook or Slack without further restriction. Ultimately, this service will enable customers to ask if they have any open bills without contacting the company.

To achieve this, we will need to create a layer between SAP Conversational AI and SAP S/4HAHA, which can be done using the SAP Cloud SDK. Hence, after this tutorial, you will have your own server running.

This article is divided into the following sections:

• Setting up OData servlet


• Setting up SAP Conversational AI


• Calling the OData service

Selecting a predefined OData service for communication

Before entering the world of OData services, it is helpful to look at previous tutorials in this series. In particular the tutorials starting with Hello World on Cloud Foundry  and up to Tutorial 4 Calling an OData Service is helpful to understand the basic structure of the application. It is advisable to not set up your server on a local host, but as a public instance. This is necessary to have the endpoints publicly available for SAP Conversational AI to access.

After setting up your initial Hello World app via the tutorial above, it is time to connect to an OData service. For the sake of simplicity, we will follow the example from Tutorial 4: Calling an OData Service. In this tutorial, one can understand how to access the business partner service and display them unformatted in a browser. This neat tutorial also shows how to deploy an OData application to Cloud Foundry.

For the purpose of this tutorial, we will use another OData service. In particular, the “Read Accounting Document” Service will be used. Please see here for the documentation. Before being able to connect to the service, it is required to set up a communication agreement for this scenario (SAP_COM_0303). This can be done through your SAP Fiori launchpad just go to the Fiori app “Communication agreements” (Note: this requires BRA role) and select “New” in the bottom left corner. A menu similar to this should open and you can just type in the code from above:



Give it a memorable and unique name, such as your user number or similar and proceed. In the next menu, you will see the services provided in this scenario. There are two different services, but the one we are interested in has the service URL ending in /API_OPLACCTGDOCITEMCUBE_SRV. You will recognize this path from the description of this service earlier. If you have problems with the steps above consult this quick reference or step 12 in the SAP Cloud SDK tutorial.

Before finishing up, a communication system needs to be specified. You can either choose an existing one or create a new one, just as you like. This should also fill the “Username” field and now the communication agreement can be saved.

To test these settings, we can amend the code given from the business partner servlet to access our accounting documents instead.

The following code will be changed:

List<BusinessPartner> businessPartners =

        new DefaultBusinessPartnerService()

                .getAllBusinessPartner()

                .select(BusinessPartner.BUSINESS_PARTNER,

                        BusinessPartner.LAST_NAME,

                        BusinessPartner.FIRST_NAME,

                        BusinessPartner.IS_MALE,

                        BusinessPartner.IS_FEMALE,

                        BusinessPartner.CREATION_DATE)

                .filter(BusinessPartner.BUSINESS_PARTNER_CATEGORY.eq(CATEGORY_PERSON))

                .orderBy(BusinessPartner.LAST_NAME, Order.ASC)

                .execute();



response.setContentType("application/json");

response.getWriter().write(new Gson().toJson(businessPartners));

We can change the file to the following to use the OData described above. The whole file looks like this. The query searches for uncleared documents and displays the company, the document number and the customer. This is later going to be extended but serves for now.

List<OperationalAcctgDocItemCube> operationalAcctgDocItemCubes =

                    new DefaultAccountingDocumentService()

                    .getAllOperationalAcctgDocItemCube()

                    .select(OperationalAcctgDocItemCube.COMPANY_CODE,

                            OperationalAcctgDocItemCube.ACCOUNTING_DOCUMENT,

                            OperationalAcctgDocItemCube.CUSTOMER)

                    .filter(OperationalAcctgDocItemCube.IS_CLEARED.eq(false))

                    .execute();



response.setContentType("application/json");

response.getWriter().write(new Gson().toJson(operationalAcctgDocItemCubes));

Accessing the URL with that path <baseurl>/AccgDocs now displays a result similar to this:

[  

   {  

      "CompanyCode":"1010",

      "AccountingDocument":"9400000020",

      "Customer":"10100001"

   }

]

This is everything we need for now. We will return to this application at a later stage.

Setting up SAP Conversational AI

The next step towards a SAP S/4HANA enabled chatbot is to create a SAP Conversational AI account (here). There are good tutorials available on the website explaining how to get started or more complex chatbots. For this deep dive it is enough to create a simple chat bot. Select New Bot in the top right. You will get to a menu like this:



It is not necessary to select predefined skills for this example, but it is an interesting addition for later projects. You can give your bot a name and give it a description, if desired. Fairly important is the last setting: You can decide to make your bot private or public. It is recommended to create a private bot.

After you created your bot, you will need to create a new Intent. Just create a new intent classed “sap_odata_tutorial” and enter the following phrases:
“Show me open bills for customer 231321”, ”open bills for customer 12342323”, “Are there any open bills for customer id 23145245”, “My customer id is 23142333, do I have any open bills?”. SAP Conversational AI might identify some entities, but you can ignore those. Rather click on the numbers and create a new entity called “customer_id”. After doing this for all, you screen should display something like in this screenshot:



SAP Conversational AI successfully identifies the customer id in the expressions, which is exactly what we want before creating our first skill.

Skills are the heartbeat of chat bots on SAP Conversational AI. They decide what happens upon user input and how to react. They are created in the “Build” tab via the “+ create skill” button on the left-hand side.



We want a skill to respond to enquiries regarding open bills, which should be the core competency (or “business”) of our bot. Simply create the skill and click on the name to adjust the settings.

You will see multiple tabs, including:

• Readme: extended description of the skill


• Triggers: under what conditions this skill is considered


• Requirements: information required beyond the trigger conditions


• Actions: what happens when the skill is executed

We will start with Triggers in this tutorial, before setting the actions. For the triggers select the box after the “If” and select “@sap_odata_tutorial” and click save. It is not required to select any of “slug”, “confidence” or “description”.



This means that this skill is only triggered, if SAP Conversational AI detects the intent “sap_odata_tutorial” in the user’s message.

The next and final step is to “add a new message group” in the Actions tab. We want to call a webhook, so we will select this option. There are now multiple options, but we want a simple post request without authorization for now. You can just paste the link used earlier <baseurl>/AccgDocs into the field and save it. Check the “body” tab to see the JSON object the Java servlet will receive when our skill is triggered.

This is everything we need to set up on SAP Conversational AI. We will now switch back to our Java application to process the information received by SAP Conversational AI.

Calling the OData Service

There are only three steps required to enable the server to process the incoming payload.

• Overwrite the Java servlet configuration to allow post requests


• A word on Authentication


• Receive the customer id and use it to query the OData service


• Send back the results

Enable Post requests

Adding this little post snipped allows to edit the Java servlet’s behavior upon post requests:

@Override

protected void doPost(final HttpServletRequest request, final HttpServletResponse response) throws ServletException, IOException {

    super.doPost(request, response);

}

This will not create any additional feature just now, but the code can be changed to handle the JSON payload.

In the standard settings the servlet requires all incoming POST requests to contain a cross-site request forgery (CSRF) preventing token to be safe for usage in web browser-originating calls. For the purpose of simplicity and because the webhook is not used from a browser, we can limit CSRF-protection to certain paths of our web application by amending the following snippet in the settings file /application/src/main/webapp/WEB-INF/web.xml. By adding /restricted in the url-pattern, you can still host CSRF protected sites, while enabling POST requests to this servlet. For more information on CSRF click here.

<filter>

    <filter-name>RestCsrfPreventionFilter</filter-name>

    <filter-class>org.apache.catalina.filters.RestCsrfPreventionFilter</filter-class>

</filter>

<filter-mapping>

    <filter-name>RestCsrfPreventionFilter</filter-name>

    <url-pattern>/restricted/*</url-pattern>

</filter-mapping>

A word on Authentication

Earlier, we chose a plain POST request for communication between our SAP Conversational AI bot and Java Servlet. To protect our Java Servlet, and ultimately the underlying data obtained via the OData service, from unintended access from sources which are not our chatbot we can use two options:

1. The validation via password and username as offered in the Authentication tab on the SAP Conversational AI website


2. Sending a token via a custom header. One can generate a random token which is checked upon access to the servlet.

Note that both options do not distinguish between different chatbot users and that chatbot security depends also on the security of the channels to which you connect your bot. Stay tuned for updates!

Calling the OData service

The incoming JSON payload needs to be parsed into a usable format. You can do this by copying the code snippet into your servlet:

static class RecastData {

    @SerializedName("nlp")

    @Expose

    private Nlp nlp;



    public Nlp getNlp() {

        return nlp;

    }

}



static class Nlp {

    @SerializedName("entities")

    @Expose

    private Entities entities;



    public Entities getEntities() {

        return entities;

    }



    public void setEntities(Entities entities) {

        this.entities = entities;

    }

}



static class Entities {

    @SerializedName("customer_id")

    @Expose

    private List<CustomerId> customerId = null;



    public List<CustomerId> getCustomerId() {

        return customerId;

    }



    public void setCustomerId(List<CustomerId> customerId) {

        this.customerId = customerId;

    }

}



static class CustomerId {

    @SerializedName("value")

    @Expose

    private String value;

    @SerializedName("raw")

    @Expose

    private String raw;

    @SerializedName("confidence")

    @Expose

    private String confidence;



    public String getValue() {

        return value;

    }



    public void setValue(String value) {

        this.value = value;

    }



    public String getRaw() {

        return raw;

    }



    public void setRaw(String raw) {

        this.raw = raw;

    }



    public String getConfidence() {

        return confidence;

    }



    public void setConfidence(String confidence) {

        this.confidence = confidence;

    }

}



static class TextResponse {

    String type = "text";

    String content;



    public String getResponse() {

        Gson gson = new Gson();

        return gson.toJson(this);

    }



    public void setContent(String content) {

        this.content = content;

    }

}



static class ResponseWrapper {

    TextResponse[] replies;

    Conversation conversation;



    public ResponseWrapper(TextResponse replies) {

        this.replies = new TextResponse[]{replies};

        this.conversation = new Conversation(new Memory());

    }



    public String getResponse() {

        Gson gson = new Gson();

        return gson.toJson(this);

    }



    public Memory getMemory() {

        return conversation.getMemory();

    }



    public void setMemory(Memory memory) {

        this.conversation.setMemory(memory);

    }



}



static class Memory {

}



static class Conversation {

    Memory memory;



    public Conversation(Memory memory) {

        this.memory = memory;

    }



    public Memory getMemory() {

        return memory;

    }



    public void setMemory(Memory memory) {

        this.memory = memory;

    }

}

These classes offer a quick solution to the transformation of the JSON data. The first four classes transform the incoming payload into a readable format and ignore every other content in the JSON object. The last four classes offer the reverse functionality and quickly transform data back into a format SAP Conversational AI accepts.

The incoming data can be transformed just like in this example below:

BufferedReader reader = request.getReader();

Gson gson = new Gson();

RecastData myData = gson.fromJson(reader, RecastData.class);

String custeomerid = myData.getNlp().getEntities().getCustomerId().get(0).getValue();

We find our customer id by calling the NLP object and the entities and then finally the customer id object. Note that this is by definition an array to accommodate multiple occurrences of entities. In this case we will only have one, hence we need to select the first one as we will only ever have one entity “customer id” in this base case.

We can then extract this value and plug it into the filter definition of the OData service call to narrow down the result accordingly. Please check the code at the end of this paragraph.

To make use of the data, we need to send back a message to the user. The best way to do this is using the prepared classes just like in the code snipped below. We will create a String depending on the result we get back. If there are no open bills (e.g. the result has size 0), that’s great! If there are, we can list all of them or, as below, just give one of them as an example. SAP Conversational AI offers other message types, such as images, buttons and quick replies, to offer a better user experience. It is possible to find these structures here.

The full code now looks like this:

package com.sap.cloud.sdk.tutorial;



import com.google.gson.Gson;

import com.google.gson.annotations.Expose;

import com.google.gson.annotations.SerializedName;

import com.sap.cloud.sdk.s4hana.datamodel.odata.namespaces.accountingdocument.OperationalAcctgDocItemCube;

import com.sap.cloud.sdk.s4hana.datamodel.odata.namespaces.accountingdocument.OperationalAcctgDocItemCubeFluentHelper;



import com.sap.cloud.sdk.s4hana.datamodel.odata.services.AccountingDocumentService;

import com.sap.cloud.sdk.s4hana.datamodel.odata.services.DefaultAccountingDocumentService;

import org.slf4j.Logger;



import javax.servlet.ServletException;

import javax.servlet.annotation.WebServlet;

import javax.servlet.http.HttpServlet;

import javax.servlet.http.HttpServletRequest;

import javax.servlet.http.HttpServletResponse;

import java.io.BufferedReader;

import java.io.IOException;

import java.util.List;



import com.sap.cloud.sdk.cloudplatform.logging.CloudLoggerFactory;

import com.sap.cloud.sdk.odatav2.connectivity.ODataException;



@WebServlet("/AccgDocs")

public class AccgDocItemServlet extends HttpServlet {



    private static final long serialVersionUID = 1L;

    private static final Logger logger = CloudLoggerFactory.getLogger(OperationalAcctgDocItemCube.class);



    @Override

    protected void doGet(final HttpServletRequest request, final HttpServletResponse response)

            throws ServletException, IOException {

        try {

            List<OperationalAcctgDocItemCube> operationalAcctgDocItemCubes =

                    new DefaultAccountingDocumentService().getAllOperationalAcctgDocItemCube().select(OperationalAcctgDocItemCube.COMPANY_CODE,

                            OperationalAcctgDocItemCube.ACCOUNTING_DOCUMENT,

                            OperationalAcctgDocItemCube.CUSTOMER)

                            .filter(OperationalAcctgDocItemCube.IS_CLEARED.eq(false))

                            .execute();



            response.setContentType("application/json");

            response.getWriter().write(new Gson().toJson(operationalAcctgDocItemCubes));



        } catch (final ODataException e) {

            logger.error(e.getMessage(), e);

            response.setStatus(HttpServletResponse.SC_INTERNAL_SERVER_ERROR);

            response.getWriter().write(e.getMessage());

        }

    }



    @Override

    protected void doPost(final HttpServletRequest request, final HttpServletResponse response) throws ServletException, IOException {

        BufferedReader reader = request.getReader();

        Gson gson = new Gson();

        RecastData myData = gson.fromJson(reader, RecastData.class);



        try {

            Entities entities = myData.getNlp().getEntities();

            String customer_id;



            TextResponse textResponse = new TextResponse();



            if(entities.getCustomerId()==null){

                textResponse.setContent("Mhm, I can't identify this customer");

            } else {

                customer_id = entities.getCustomerId().get(0).getValue();

                List<OperationalAcctgDocItemCube> operationalAcctgDocItemCubes =

                        new DefaultAccountingDocumentService().getAllOperationalAcctgDocItemCube().select(

                                OperationalAcctgDocItemCube.COMPANY_CODE,

                                OperationalAcctgDocItemCube.ACCOUNTING_DOCUMENT)

                                .filter(OperationalAcctgDocItemCube.CUSTOMER.eq(customer_id))

                                .execute();



                if (operationalAcctgDocItemCubes.size() > 0) {

                    OperationalAcctgDocItemCube oneDoc = operationalAcctgDocItemCubes.get(0);

                    textResponse.setContent("There are uncleared documents. For example document " + oneDoc.getAccountingDocument() + " with company " + oneDoc.getCompanyCode());

                } else {

                    textResponse.setContent("There are no uncleared documents! That is great");

                }

            }



            ResponseWrapper responseWrapper = new ResponseWrapper(textResponse);

            response.setContentType("application/json");

            response.getWriter().println(responseWrapper.getResponse());



        } catch (final ODataException e) {

            logger.error(e.getMessage(), e);

            response.setContentType("application/json");

            response.getWriter().write(e.getMessage());

        }

    }



    static class RecastData {

        @SerializedName("nlp")

        @Expose

        private Nlp nlp;



        public Nlp getNlp() {

            return nlp;

        }

    }



    static class Nlp {

        @SerializedName("entities")

        @Expose

        private Entities entities;



        public Entities getEntities() {

            return entities;

        }



        public void setEntities(Entities entities) {

            this.entities = entities;

        }

    }



    static class Entities {

        @SerializedName("customer_id")

        @Expose

        private List<CustomerId> customerId = null;



        public List<CustomerId> getCustomerId() {

            return customerId;

        }



        public void setCustomerId(List<CustomerId> customerId) {

            this.customerId = customerId;

        }

    }



    static class CustomerId {

        @SerializedName("value")

        @Expose

        private String value;

        @SerializedName("raw")

        @Expose

        private String raw;

        @SerializedName("confidence")

        @Expose

        private String confidence;



        public String getValue() {

            return value;

        }



        public void setValue(String value) {

            this.value = value;

        }



        public String getRaw() {

            return raw;

        }



        public void setRaw(String raw) {

            this.raw = raw;

        }



        public String getConfidence() {

            return confidence;

        }



        public void setConfidence(String confidence) {

            this.confidence = confidence;

        }

    }



    static class TextResponse {

        String type = "text";

        String content;



        public String getResponse() {

            Gson gson = new Gson();

            return gson.toJson(this);

        }



        public void setContent(String content) {

            this.content = content;

        }

    }



    static class ResponseWrapper {

        TextResponse[] replies;

        Conversation conversation;



        public ResponseWrapper(TextResponse replies) {

            this.replies = new TextResponse[]{replies};

            this.conversation = new Conversation(new Memory());

        }



        public String getResponse() {

            Gson gson = new Gson();

            return gson.toJson(this);

        }



        public Memory getMemory() {

            return conversation.getMemory();

        }



        public void setMemory(Memory memory) {

            this.conversation.setMemory(memory);

        }



    }



    static class Memory {

    }



    static class Conversation {

        Memory memory;



        public Conversation(Memory memory) {

            this.memory = memory;

        }



        public Memory getMemory() {

            return memory;

        }



        public void setMemory(Memory memory) {

            this.memory = memory;

        }

    }



}

This can now be pushed to SAP Cloud Foundry and we can test the service on the SAP Conversational AI website. Just click on the chat button in the bottom right and message the bot with one of the expressions defined in the intent section. Remember from earlier that customer 10100001 (Note: might be different in your case) had an open bill, so let’s ask the bot about customer 10100001:



We see that the bot responds immediately with the style we defined before. The other customer id doesn’t have any open bills and the bots also responds accordingly.

This is how to create a SAP Conversational AI Bot using the SAP Cloud SDK to enrich messages. It is perfectly possible to create a more complicated bot from here, by teaching it more intents and expressions, add additional filters and create a more sophisticated flow of information. You could start by amending the parameters reported back, choosing the due date for example. Further, it is possible to deploy the bot to Twitter or Facebook and allow customers to get a live update on their current bills with the company. Then you might start leveraging the memory to handle the situation that the user doesn’t mention his id right away: (U: ‘Do I have open bills?’ Bot: ‘Sorry, what is your customer id?’ …). Go and simplify support!