- SAP Managed Tags:
So you decided to leave your IT job and sell your home-made candles online?
Chances are you are doing so through an online business platform such a Shopify.
But as you business grows, you need to integrate your Shopify shop with other systems such as an ERP. And this is my job: explaining you how to do so!
This can be achieved using SAP BTP Integration Suite and we'll see how to do that in this blog, especially how to connect the Shopify Webhook to a Cloud Integration integration flow.
In this blog, we will assume that you know the SAP Business Technology Platform (BTP) and the Integration Suite - a least from a basic perspective.
Setting the scene
Cloud-native solutions such as Shopify often provide webhooks which let you call a REST API when a specific event happens. This is perfect to call an integration flow that will process the Shopify data.
However, webhooks are not very good at security. Hence we will need to use both API Management and Cloud Integration to implement the pilot.
What do we need?
First of all, the scenario will need data massage.
Indeed, the data coming from Shopify cannot be used "as-is" in S/4HANA since the data structure is completely different. This is done in an integration flow in the Cloud Integration capability (but will not be part of this blog). Also, if you connect to another system than S/4HANA, you may need connectivity using out-of-the-box Cloud Integration Adapters.
Secondly, the scenario will need specific security features.
Indeed, Shopify webhooks have no way to authorize against the target API and the webhook sends the full payload when triggered. API Management capability can come to the rescue: it sits between the consumer and the API implementation (ie. Shopify webhook and integration flow) and can receive non-authorized requests and perform further security checks.
Let's implement all that in the next steps!
Technical implementation
Setting up Cloud Integration
The integration flow that will be called by Shopify's webhook will be very simple: it will be exposed as REST API and will eventually contain the mapping to the backend system (eg. S4).
Start by creating an integration flow in the package of your choice. I created a dedicated package for Shopify for instance.
As you may know, to call an integration flow, you need to have the right credentials and roles to do it.
An integration flow can be called using basic authentication, with a check on the caller's role. This is fine for a pilot, hence we'll use that as security mechanism.
Add an HTTP Sender Adapter with the configuration as depicted below.
Deploy your integration flow so we can test it later on.
To call that integration flow, you will need to create an API Service key that will contain credentials (clientId & clientSecret) for a specific role, such as "ESBMessaging.send".
You may have created an API Key: in that case you can use its credentials. If not, follow the next steps.
The whole process of creating an API key is described in detail here, but basically you need to:
- go to your subaccount
- create a new instance of SAP Process Integration Runtime ("integration-flow" plan)
- create a service key for that instance
The result looks somewhat like this:
Once the key is created, copy clientId and clientSecret: we will need it later to trigger the iflow.
Setting up Postman for testing
I am a big fan of Postman since its very beginning, so I will use that to test the call against my integration flow. But you can also use free tools such as ReqBin.
Before you create the request, we need to get the endpoint URL of the integration flow. It can be found in the Monitor / Integration and APIs / Manage Integration Content screen.
Copy the URL of the endpoint and use it in the client of your choice to create your POST request.
In the Authentication tab, select Basic Authentication and enter your clientId and clientSecret as credentials.
It should look like that.
Setup Shopify Webhook
Let's now define a webhook in Shopify. To do so, login to Shopify and navigate to the Settings (the link is at the bottom left of your screen).
Now select Notifications / Webhooks.
Create a webhook for order creation and insert the URL of your integration flow.
Also, copy the value of the key since we will need it later.
Don't get too excited though: this won't work (test it if you want: ... / Send Test) because the integration flow requires authentication!
A way to enable that, is to use API Management, to inject the credentials there.
Setup API Management
The main reason behind using API Management is to centralize security, traffic management and analytics of APIs - making the management of your IT landscape easier.
This works mainly through the usage of so called API Management API Proxies which sit between the caller and the implementation of the API.
To create an API proxy, navigate to your Integration Suite, Configure / APIs.
Click on Create and fill in the fields as depicted below.
The URL should be the endpoint URL of your iFlow.
Once the API Proxy is created, click on the Key Value Maps tab on the top.
Indeed, in order to safely inject the Cloud Integration credentials, we will use the Key Value Map feature of API Management: you can store values in an encrypted way, available only to the API proxy during runtime (vs. hardcoded in the API proxy itself).
Create 2 encrypted KVMs named Shopify and Cloud Integration.
In the Shopify KVM, create a key named WebHookKey containing the value of the Shopify key you copied before.
Do the same for CloudIntegration, with the clientId and clientSecret values from your Cloud Integration API Service key.
Go back to your previously created API proxy and click on Policies (top right on your screen).
Create the following policies to secure the API Call (at least a bit).
1- KeyValueMapOperations policy: retrieve the value of the Shopify key from the KVM. This policy will set the KVM value in a private variable (not exposed through logs or in debug mode).
<KeyValueMapOperations mapIdentifier="Shopify" async="true" continueOnError="false" enabled="true" xmlns="http://www.sap.com/apimgmt">
<Get assignTo="private.Shopify.webhookkey">
<Key>
<Parameter>WebHookKey</Parameter>
</Key>
</Get>
<Scope>environment</Scope>
</KeyValueMapOperations>
2- KeyValueMapOperations policy: same as above, retrieve the value of the Cloud Integration credentials from the KVM.
<KeyValueMapOperations mapIdentifier="CloudIntegration" async="true" continueOnError="false" enabled="true" xmlns="http://www.sap.com/apimgmt">
<Get assignTo="private.clientId">
<Key>
<Parameter>clientId</Parameter>
</Key>
</Get>
<Get assignTo="private.clientSecret">
<Key>
<Parameter>clientSecret</Parameter>
</Key>
</Get>
<Scope>environment</Scope>
</KeyValueMapOperations>
3- BasicAuthentication policy: creates a base64-encoded Authorization header, ie. basic authentication, for the integration flow, using the previously retrieved clientId and clientSecret from the KVM.
<BasicAuthentication async='true' continueOnError='false' enabled='true' xmlns='http://www.sap.com/apimgmt'>
<Operation>Encode</Operation>
<IgnoreUnresolvedVariables>true</IgnoreUnresolvedVariables>
<User ref='private.clientId'></User>
<Password ref='private.clientSecret'></Password>
<AssignTo createNew="false">request.header.Authorization</AssignTo>
</BasicAuthentication>
4- Python Script policy: executes a hash check. Indeed, we want to check that the call originated from Shopify and was not altered. This is done through the verification of the HMAC (Hash Message Authentication Code) sent from Shopify in a header, using the key available only to Shopify admins (in the encrypted KVM).
<Script async="false" continueOnError="false" enabled="true" timeLimit="200" xmlns='http://www.sap.com/apimgmt'>
<ResourceURL>py://HashCheck.py</ResourceURL>
</Script>
Now add a new script on the left-hand side menu and name it just like the one you are referencing in your policy, eg. HashCheck.py.
import hashlib
import hmac
import base64
import random
import string
def make_digest(secret, payload):
hashBytes = hmac.new(secret, msg=payload, digestmod=hashlib.sha256).digest();
base64Hash = base64.b64encode(hashBytes);
return base64Hash;
message = "";
message = flow.getVariable("request.content");
webhookHash = flow.getVariable("request.header.X-Shopify-Hmac-Sha256");
resultHash = make_digest( flow.getVariable("private.Shopify.webhookkey"), str(message) );
flow.setVariable("resultHash",resultHash);
lettersAndDigits = string.ascii_letters + string.digits;
randomKey = ''.join((random.choice(lettersAndDigits) for i in range(44)));
flow.setVariable("randomKey",randomKey);
doubleWebhookHash = make_digest( randomKey, webhookHash );
doubleResultHash = make_digest( randomKey, resultHash );
flow.setVariable("doubleWebhookHash",doubleWebhookHash);
flow.setVariable("doubleResultHash",doubleResultHash);
if (doubleResultHash != doubleWebhookHash):
raise NameError("Invalid Origin! Different HMAC.")
Update, save and deploy the API Proxy.
Now copy the URL of the API Proxy from the overview.
If you test the API Proxy through Postman, you will see that API Management is preventing you to do so since the Hash-Check was unsuccessful. Indeed: both the HMAC-SHA header as well as the payload are missing.
This is why we now test it in situ, using Shopifiy's webhook.
Update Shopify
Go back to your Shopify account and update the WebHook URL with the API proxy URL.
If you perform a new test, you should now see the integration flow being triggered properly:
If you have time, you can also intercept the webhook call from Shopify through API Management Debug, and recreate the call in Postman using the X-Shopify-Hmac-Sha256 header value and the payload. This call will work, but as soon as you play the "man-in-the-middle", ie. as soon as you alter the payload, you will get an error.
Mapping in the integration flow
In order to properly communicate to the backend system, you need to perform a mapping between incoming Shopify message and the expected S/4HANA message structures.
This is done using the message mapping flow step, but very much depends on your own setup. Hence I am not going through the details.
However, note that you will need to create an OpenAPI specification from the Shopify webhook call since it is not provided in the documentation. Also, AFAIK, there is no documentation of the data structure of an order, at least not in a usable YAML or JSON format. Feel free to comment if I have overseen anything!
For the OpenAPI spec, the steps I took were:
- Get the payload from the Webhook APICall (Debug of API Management for instance)
- Transform the payload into a JSON data structure
https://roger13.github.io/SwagDefGen/ - Create an OpenAPI file and inject that data structure in the schema part of the OpenAPI definition
https://editor.swagger.io/
When importing that OpenAPI specification into the message mapping flow step of your integration flow, you will notice that some fields could not properly be translated into a proper structure because of missing values.
I replaced the empty properties with dummy properties, enough for the pilot."note_attributes": {
"type": "array",
"items": {
"type": "object",
"properties": { }
}
},"note_attributes": {
"type": "array",
"items": {
"type": "object",
"properties": {
"note_attribute_property": {
"type": "string",
"description" :"Dummy note attribute property"
}
}
}
},
Conclusion
Using the SAP BTP Integration Suite, we have quite quickly created the communication between a cloud application webhook and an S/4HANA system.
As said before, this is only a pilot and a production-grade setup would need additional features such as traffic management or IP filtering, as well as a mapping, possibly more processing in the integration flow and a better security concept.
Best regards!
Sven
