Introducing the REST API Provider in SAP Landscape...

naeem_maqsud · ‎2022 Nov 30

Introduction

In LaMa Enterprise Edition SP25, we have added a provider definition for calling REST APIs. We have supported inbound REST API calls since SP3 but outbound API calls were available with the HTTP provider definition with limitations (POST requests only, had a limited response parsing and support for only basic authentication). The new feature is much more advanced as you will learn in this blog.

In a previous blog I described a workaround to make outbound REST API calls via the "script registered with a host agent" provider definition. I showed a simple example of calling Ansible but did not make REST calls and rather relied on the CLI interface of Ansible. In this blog I will show how Ansible and other applications can be called directly from LaMa using the new REST API provider.

The REST provider is part of automation studio in LaMa, and therefore various other providers can be combined to create some really powerful automation workflows.

Feature: REST API (Outbound)

First, lets get some understanding of the new feature. LaMa can now make direct RESTful calls to any application/service that supports REST API. In other words LaMa acts as a REST client.

Note: Some applications/services may not support REST API but can still receive and respond to REST calls from LaMa. For example Slack API is not RESTful but we can still communicate with it using the REST provider in LaMa.

Figure 1: New Outbound REST API Feature

The new REST API provider can be considered as the successor of the HTTP provider definition, can parse any JSON response and extract specific properties. It is also possible to construct a request body using a templating engine and set header entries of any kind.

Methods supported: GET, POST, PATCH, PUT and DELETE

Response parsing: Currently supports JSON only

Authentication: No Auth, API Key, Bearer Token, Basic Auth, OAuth2

When choosing the provider type of REST, you then have options of provider subtypes of Standalone, Token Supplier, and Result Retriever.

Standalone Subtype: This can be considered as the main provider that is used when creating custom operations or processes. It can not be used by other provider definitions.

Token Supplier Subtype: This is a provider to retrieve a dynamic token that is used for authentication. This can be configured for use by other REST provider definitions such as the REST Standalone.

Result Retriever Subtype: This provider handles a long running operation by having a separate "start" and "get result" call. Thus when configured for use by other provider definitions, it can run multiple times until the operation completes. The sleep time is configured to define the interval of running the result retriever.

Token types can be selected as:

Token: Used when the token is long lived and can be entered directly into the provider definition where it is saved in an encrypted form.

Dynamic token: This is used when the token has a short life time and is retrieved automatically before each request. It can be used for OAuth2 based authentications.

Figure 2: GUI Selection Options for Provider Subtypes, Authentication. and Methods

Additional configuration settings that are covered in this blog are explained.

Parameters: This is where parameters used by the provider definition can be stored such as username and password (encrypted)

Request Configuration: Used for adding header entries (e.g. application/x-www-form-urlencoded)

Request Body: The JSON data sent when using POST / PUT / PATCH.

Optionally, Apache FreeMarker templates can be used in the body and LaMa will detect and interpret this. This gives you a lot of flexibility when handling changing data, logical/math operations, formatting etc. When using parameters use "_" instead of "-". For example a parameter that would normally have a prefix such as PARAM-xx or TRANS-xx should be written as PARAM_xx and TRANS_xx respectively.

Response Parsers: One or more parsers can be added using the JMES Path expression (query language for JSON)

If the JMES Path of any response parser can not be resolved in the response, the operation will fail (not of type ERROR, WARNING or INFORMATION)

Response Parser types are listed here in the LaMa Guide

Figure 3: Additional Configuration Tabs

The REST API provider definition can be used for:

Custom operations (required entities selected in the Operation Uses section)

Custom hooks (required entities selected in the Hook Uses section)

Custom provisioning processes (Provisioning Replacement checkbox marked)

In addition to making API calls to other applications, LaMa can also make REST API calls to itself. As noted earlier, inbound REST API support has been available for several years and this can be leveraged to more easily incorporate into custom workflows that involve making GET, POST, PATCH, PUT and DELETE calls to LaMa itself.

Figure 4: RESTful Calls to Itself

With the new REST API provider feature (outbound), integrations to other tools/applications can be built into LaMa giving flexibility and additional automation capabilities when managing SAP system landscapes. The diagram below shows a few of the possible integrations.

Figure 5: Potential Integration with 3rd Party Apps

Use Case Examples

To better understand the benefits of this new feature, let me illustrate a couple of scenarios. There can be numerous other use cases where LaMa can orchestrate operations involving built-in functionality and external tools via direct API calls. But we can not cover everything so lets just focus on a few.

Operating System Level Maintenance

If your organization uses Ansible Tower or AWX for OS-level automation (OS patching, password changes, OS package install etc.) it is now much easier to integrate this into LaMa for a one-click operation. If you also use Slack or Teams, notifications can be sent to specific channels related to the operation.

The diagram below depicts an example where a custom OS patching operation makes REST API calls to Ansible Tower (or AWX) and Slack (or Teams) to trigger OS patching and also post messages about the operation. SAP systems are stopped and started as part of the fully automated process.

Figure 6: REST API Calls to Ansible and Slack from LaMa

Click to enlarge diagram

Backup & Restore

LaMa has built-in support for using 3rd party backup tools to perform a restore based refresh, but there are times when a customer needs to trigger backups or restores from LaMa unrelated to a provisioning operation.

Prior to SP25, there was the option to use "script registered with host agent" provider and now it can be done more easily by making REST API calls directly to the 3rd party backup tool. Let's assume that a customer wants to take a backup before performing maintenance. This can either be a standalone operation or can be added as a hook to an existing or other custom operation, If there is a need to perform a restore, that can also be triggered from LaMa.

Figure 7: LaMa Interaction with 3rd Party Backup Tools

Cloud Provisioning - Example 1

Let's now consider an example that would apply in a cloud environment as another way to understand the new functionality.

This is an example of a scenario in cloud provisioning to perform system copies when there are no existing target servers. In this case a custom provisioning process can be created that combines the built-in system copy operation with the provisioning of empty virtual machines.

Figure 8: REST API Calls to Ansible and Slack from LaMa

Click to enlarge diagram

Some of the cloud adapters for LaMa also support the provisioning of empty VMs when performing a system copy or system clone procedure in LaMa. For example the Azure Connector for LaMa allows the option to provision empty VMs using the Azure Resource Manager. However, the above example gives more flexibility to choose other Infrastructure as Code (IaC) tools and support for multiple clouds.

Cloud Provisioning - Example 2

One more example of a scenario in cloud provisioning that I would like to cover is LaMa triggering the provisioning of vanilla installations of SAP applications (including the cloud infrastructure), discover the newly created system and start managing with LaMa. All of this would be a single custom operation that relies on API calls.

Side Note - Fully Automated Deployment of Production Ready S/4HANA with CAL

It is prudent to mention that another SAP solution (sister product of LaMa known as Cloud Appliance Library) is available that deploys a complete and customized production ready S/4HANA landscape in the cloud. This allows a customer to use their own custom stack from Maintenance Planner. In addition High Availability can be optionally selected. Refer to this blog for more info.

The diagram below shows that API calls are made first to provision the infrastructure followed by calls to Ansible Tower or AWX to launch a playbook for installation of SAP. The SAP software has already been downloaded to a repository in the cloud (e.g. blob storage) that can be accessed by the playbook. The custom process also makes REST API calls to itself to discover the new systems and in the end a Teams message is posted.

Figure 9: Example Workflow for SAP Application Install via API calls from LaMa

Click to enlarge diagram

LaMa REST API call samples

Now I will go over some of the REST API calls covered above. This is not a how-to blog so the creation of a custom process/operation will not be covered in detail. Instead I will show some examples of making calls to Ansible AWX and Slack that can then be used to create custom processes like the above use cases.

Disclaimer

This blog is published “AS IS”. Any software coding and/or code lines / strings (“Code”) included are only examples and are not intended to be used in any productive system environment. The Code is only intended to better explain and visualize the features of the SAP Landscape Management Automation Studio. No effort has been made to make the code production quality (e.g. security, error handling, robustness, etc). If you use any of the code shown, you are doing it at your own risk.

Information in this blog could include technical inaccuracies or typographical errors. Changes may be periodically made.

Assumptions

It is assumed that your environment is already setup correctly for creating custom processes in LaMa. This blog does not go over all the configuration steps. If this is your first time using LaMa Automation Studio then I recommend that you start with something simple. Please refer to this blog for other examples. All examples below use an environment with LaMa Enterprise Edition 3.0 SP25

REST Calls to Ansible Tower / Ansible AWX

This is a simple example to demonstrate a RESTful call to Ansible and applies to either Ansible Tower (commercial version) or Ansible AWX (open source version).

I will use a simple playbook that downloads an rpm for the figlet utility, installs the rpm, uses the figlet command and then removes the utility.

The sample playbook figlet_rpm.yml file can be accessed at GitHub for use in Ansible Tower or Ansible AWX.

The Provider Definitions can also be downloaded from GitHub or you can follow the steps below to create them.

Note: When importing the XML file you must ensure that you make the necessary changes in the provider definitions to use the correct URLs and credentials. The XML file uses dummy URLs and empty fields for credentials.

The steps to follow in order to create a custom operation to execute the playbook from LaMa:

Create the Provider Definition that retrieves the Ansible AWX token and acts as supplier to another provider definition

Set the provider type, subtype, URL, authentication type, username, password, and method as shown below:

Figure 10: Screenshot - Provider Type, Subtype, URL, Authentication Type, Username, Password, and Method

Click to enlarge diagram

Edit the above provider and fill in the Header Entry

Figure 11: Screenshot - Header Entry

Click to enlarge diagram

Enter the Request Body

Figure 12: Screenshot - Request Body

Click to enlarge diagram

Enter the JMES path in the Response Parser tab

Figure 13: Screenshot - Response Parser with JMES Path

Click to enlarge diagram

Create Provider Definition for obtaining the Ansible AWX job results

Set subtype as Result Retriever, give URL (note that "<REF-jobLocation>" will be filled by the Standalone provider that runs the ansible job - see next section), set authentication to DYNAMIC TOKEN as it is provided by the token supplier (above), token type is BEARER, select the token supplier provider definition we created earlier, and method is GET.

Figure 14: Screenshot - Provider Type, Subtype, URL, Authentication Type, Token Type, Token Supplier, and Method

Edit the provider definition and add Response Parsers

Figure 15: Screenshot - Response Parsers

Click to enlarge diagram

Create Provider Definition for executing the ansible playbook

Set subtype to Standalone, URL for AWX (change 11 to the ID of your template), authentication is DYNAMIC TOKEN as we use a token supplier, specify the token supplier created earlier, specify the result retriever created earlier, operation will be against instance and method is POST.
- To get the ID of your template, you can put the following URL in a web browser: http://<AWX host>:<AWX port>/api/v2/job_templates

Figure 16: Screenshot - Provider Type, Subtype, URL, Authentication Type, Token Type, Token Supplier, Result Retriever, and Method

Edit the provider definition and add parameters

Figure 17: Screenshot - Parameters

Click to enlarge diagram

Add Request Body. This limits the execution to the instance listed. The variable translates to the instance hostname that is selected in LaMa for the execution

Figure 18: Screenshot - Request Body

Click to enlarge diagram

Add Response Parsers. Note the jobLocation which is what will be used by the "My AWX job results" provider to fill in the URL

Figure 19: Screenshot - Response Parsers

Click to enlarge diagram

Create the Custom Operation

In order to execute the REST API call to Ansible AWX and execute the playbook, we need to create a custom operation from the provider definitions created earlier.

We need only one of the above three provider definitions "My Execute AWX template" as this already includes the other two.

Figure 20: Screenshot - Custom Operation Creation

Click to enlarge diagram

Now the above custom operation can be tested from any instance. The Linux utility "figlet" will be downloaded and installed, figlet command executed, and the utility deleted. You will be able to see the output in AWX logs.

API Calls to post messages to Slack

I now want to be able to post a Slack message as part of a custom process. So first I will create a provider definition and then follow with custom operations. I will create two operations where the first will be for sending a static message (e.g. start of operation) and the second will allow message to be modified.

Slack provider definition

As I can use a long lived token, I can indicate authentication of TOKEN and token type of Bearer with a value

Figure 21: Screenshot - Provider Type, Subtype, URL, Authentication Type, Token Type, and Method

Add parameter

Figure 22: Screenshot - Parameters

Click to enlarge diagram

Add Request Body. Note the use of curly brackets for the parameters which will be interpreted by the Apache FreeMarker template engine in LaMa. TRANS_stdOut which will be transferred from the preceding operation and PARAM_Message that will be filled via the parameters supplied. Since curly brackets are used, LaMa uses the template engine and in case the parameters come back as null, a default value is assigned. For more explanation of handling nulls please refer to this link.

Figure 23: Screenshot - Request Body

Click to enlarge diagram

Optionally we can add a response parser to handle basic errors

Figure 24: Screenshot - Response Parser

Click to enlarge diagram

Create the Custom Operations for Slack

Lets first create a custom operation called "Post to Slack BEGIN" that uses the above provider definition.

We then add a parameter with a custom default value:

Figure 25: Screenshot - Parameter (Custom Default Value)

Click to enlarge diagram

Now we create the second custom operation called "Post to Slack CUSTOM" that is the same as above except it does not have a default value. This will allow us to modify the message we post.

Now both of the above can be tested by triggering from the operations menu and any instance. You can verify that the slack message was posted.

The custom operations XML file for import can be accessed at GitHub

Note: You must ensure that you make the necessary changes in the provider definitions to use the correct URLs and credentials. The XML file uses dummy URLs and empty fields for credentials.

Custom Process Example

Now that we have a few custom operations for REST API calls, lets now put them together to create a custom process that simulates the first use case discussed. Instead of patching the operating system we will install a Linux utility (figlet). In addition, to keep it simple we will only shut down an application instance rather than the whole system.

Create the custom process as shown below using the built-in stop and start operations as well as the custom operations created earlier.

Figure 26: Screenshot - Custom Process Steps

Name: Patch Demo

Entity Type: Instance

Group: Blog Demo

The Operation Parameters (username, password) for "My Execute AWX template" should be selected

For "Post to Slack BEGIN" leave the Parameter (Message) unchecked

For "Post to Slack CUSTOM" select the Parameter (Message) and also define a Custom Default Value. At time of execution you can change it or leave it as is

Figure 27: Screenshot - Parameter (Custom Default Value)

Click to enlarge diagram

Now the new custom process can be triggered from LaMa:

Operations -> Systems -> Instance -> Blog Demo -> Patch Demo

Enter the username and password of Ansible AWX (reminder: this is used to authenticate and retrieve the token)

Accept the default message for Slack or change it

Set and Execute

Figure 28: Screenshot - Completed Steps

Click to enlarge diagram

First Slack post shows a URL without a job id and the second one includes it as it received the information from the custom operation that executed the ansible template (playbook).

Clicking on the URL will show the output of the playbook run.

The custom process XML file for import into LaMa can be downloaded from GitHub

Note: You must ensure that you make the necessary changes in the provider definitions to use the correct URLs and credentials. The XML file uses dummy URLs and dummy/empty fields for credentials.

Conclusion

What has been shown in this blog is just the tip of the iceberg in terms of what can be done with the new feature. There is an increased demand for integration with other tools and use of APIs is the primary mechanism, and now it can be done in a much better way with direct API calls from LaMa. With the XML files on GitHub, you can get started quickly and get a feel of the feature.

If you have any question or would like to provide feedback please feel free to use the comment section.