Splunk - Part 1 : SAP APIM Logging & Monitoring

[santhosh_kumarv]

The OWASP Foundation's API Security Project lists Insufficient Logging & Monitoring as one of the topmost critical API security Risks. Insufficient logging and monitoring, coupled with missing or ineffective integration with incident response, allows attackers to take advantage and fully compromise systems. So it's important to have the right logging and monitoring solution in place.

SAP API Management has Message Logging Policy to send Syslog messages to third-party log management services. There is a blog already to demonstrate logging to Loggly using it. Message Logging Policy communicates to the log service using TCP or UDP Transport protocol.

In this blog, we will see how to log API interactions to Splunk Cloud using the HTTP Application Protocol. The next blog will focus on using Splunk for SAP CPI, precisely for MPL.

Why HTTP(TLS) and not TCP/UDP?

Splunk for security reasons accepts TCP/UDP connections only from configured forwarders that are installed in the Application that require integration. It is not possible to Install a Splunk Forwarder in SAP API Management (SAP Cloud Platform) and hence we cannot use TCP/UDP Protocol. So the Message Logging Policy is obsolete for Splunk Cloud/Enterprise use case.

Logging Solution

Splunk has a Data Input type called HTTP Event Collector (HEC) which is a fast and efficient way to send data to Splunk Enterprise and Splunk Cloud over HTTP (or HTTPS) Application Protocol. We will use a Service Callout from SAP API Management to log the Events to Splunk Cloud.

About the HTTP Call to Splunk

• The Target URL should be the HTTP Event Collector API


• The Authorization Header should contain Splunk <HEC_Token>


• The Payload should be in the below format when using the JSON Source type.
  

  {
  
  	"time": 1589134706278, // timestamp in Unix/epoch format
  
  	"index": "sapapim_dev", // Index Created in Splunk
  
  	"host": "dev.apim.com", // A host to identify the event
  
  	"sourcetype": "_json", // Use JSON standard Source Type
  
  	"source": "apim.logs", // A source type value of identification
  
  	"event": {
  
  		"your_Event_Name": "your_Event_Value"
  
  	}
  
  }​

  
  

Implementation

Below are the high-level steps to implement Splunk Logging from SAP API Management.

1. Create an Index in Splunk for SAP APIM


2. Create an HEC and associate the Index. This yields an HEC Token.


3. Enable the HEC Token


4. Implement Service Callout in SAP APIM for logging Events.

1 Splunk - Create Index

Log in to Splunk Cloud and open Settings --> Data --> Indexes

Create an Index of your choice. Eg:sapapim_dev

2 Splunk - Create HEC

Open Settings --> Data --> Data Inputs and +Add New  HTTP Event Collector

Enter a Name for the Event Collector

Complete the Input Setting by selecting Source type as _json and App.
Associate the Index created for SAP APIM with this HEC.
Review the Configuration and Submit.
It will yield an HEC Token.

3 Enable HEC Token

Open Settings --> Data Inputs --> HTTP Event Collector -->Global Settings and enable All Tokens.
Enable the token for the HEC just created if not Enabled already.

4 SAP APIM - Service Callout

Open the API Proxy that you need Splunk Logging to be implemented. Add a Service Callout policy in the flow as per your requirement.
In my case I want the request and response payload to be logged to Splunk and hence I have added the Service Callout Policy to Proxy Endpoint --> PostFlow --> Outgoing Response.
Note: Cannot be added to Post Client Flow as it allows only Message Logging policy.
Below is the Policy Code Snippet.

<ServiceCallout async="true" continueOnError="true" enabled="true" xmlns="http://www.sap.com/apimgmt">

	<Request>

		<Set>

			<Headers>

				<Header name="Content-Type">application/json</Header>

				<Header name="Authorization">Splunk e43b4b64-xxxx-xxxx-xxxx-xxxxxxx</Header>

			</Headers>

			<Payload contentType="application/json" variablePrefix="#"

        variableSuffix="@">{

	"time": #client.received.start.timestamp@,

	"index": "sapapim_dev",

	"host": "dev.apim.com",

	"sourcetype": "_json",

	"source": "apim.logs",

	"event": {

		"APIProxy": "Get_OAuthToken",

		"requestVerb": "#request.verb@",

		"ClientStatus": #message.status.code@,

		"reqMessage":"#request.content@",

		"resMessage":#response.content@

	}

}</Payload> 

			<Verb>POST</Verb>

		</Set>

	</Request>

	<!--Remove Response Element if you want Service Callout work Asynchronous--> 

	<Response>splunk.response</Response>

	<Timeout>30000</Timeout>

	<HTTPTargetConnection>

		<URL>https://input-instance.cloud.splunk.com:8088/services/collector/event</URL>

	</HTTPTargetConnection>

</ServiceCallout>

When this policy is executed in the Outgoing Response Flow or Fault Rule or wherever you have added to, a JSON payload in the Splunk required format with API interaction details are logged and indexed in Splunk.

SAP API Management - DIY

• Read my earlier blog SAP API Management – Handling Faults using FaultRules and DefaultFaultRule to know how to add the policy to the Default Fault rule. Use this approach to attach the Service Callout Policy to log the Errors to Splunk.


• Read my blog SAP API Management – API Proxy Troubleshooting Technique or SAP Help Variable Reference to know the runtime variables that can be used for logging.


• You can also create this Splunk logging as a Shared Flow as shown in this blog SAP APIM : Shared Flow and call from other API Proxy Asynchronously by removing the <Response> element from the Service Callout policy.

Result

• 
  
  
  
  • The events can be used to draw Reports and Dashboards
    
  
  
  • The events i.e. API Interactions that are logged in Splunk and can be searched for troubleshooting or incident analysis.
    
  
  
  • The events can be configured to Send Alert Email/ Create Jira Ticket / Send to Webhook etc in realtime or periodically.
  
  
  
  

Once you have the data, the result is purely dependent on the capability of the platform and your creativity in utilizing it 🙂

[pandey_anuj]

Hi Santosh,

Nice blog.. You are using service callout for shipping logs however SAP APIM has built in policy called message logging for the same purpose.Is there a reason to use service callout in your example except it is shipping logs over HTTP?

Thanks,

Anuj

[santhosh_kumarv]

Hi Pandey, Yes. Message logging policy communicates over TCP/UDP, however, Splunk requires special set-up for those protocols. This is why logging is implemented using TLS. I have detailed out this already in the Blog under "Why HTTP(TLS) and not TCP/UDP?".

Thanks
Santhosh.

[senthilsubram]

santhoshkumar.vellingiri  Thank you this detailed blog. We followed it and were able to make it work with SPLUNK. Everything works as long as the flow is successful but if the flow fails, say for example on a security policy then the flow goes to error state and message call out policy is not executed to log into SPLUNK. Can you help us to understand if it is possible to log the errors on SPLUNK and how? You advice would be much appreciated.

[santhosh_kumarv]

ssubra17 Yes. You can follow my blog SAP API Management – Handling Faults using FaultRules and DefaultFaultRule to implement fault handling and include the service callout in it. It's working like charm for almost 2.5 years in my implementation. Good Luck!

[senthilsubram]

Thank you santhosh.kumarv  for your quick reply. We followed this blog and using fault rule sequence to log into SPLUNK but the problem is reusability. Every time we create API proxy, we need to download and make changes to the XML file. We are looking for something like Policy templates  which we know is not possible here but some sort of templates. Any advise on alternate solution ?

[former_member795826]

Hi Santhosh,

Just one question I have. Is this setup applicable to SAP cloud products? I have to onboard SAP audit logs of SAP cloud to my Onprem Splunk. Just curious if this is possible. Your reply would help me to move further. Thanks.

[siddharthmatalia9]

Hello Santhosh,

 

We followed the process the way it has been asked in the blog but still we are unable to see the data into splunk, we are ingesting the data into Heavy Forwarder and from Heavy Forwarder it get forwarded to splunk clould.

I have attached the API logs, please check and suggest.

[martinbuselmeier]

Hi Raja,

did you manage to connect your OnPrem Splunk instance? I've got the same problem, we want to connect our SAP API Management Instance to our OnPrem Splunk instance. Wondering if we can use the SAP Cloud Connector or if we need to find another solution

[Alain_P]

Hi @santhosh_kumarv ,                                                                                                                                   Thank you for this nice blog. Please what variable are you using to calculate the API Processing time?