Using the Streaming Web Service (SP11)

[Former Member]

Part A: Configuring the Streaming Web Service for SAP HANA Smart Data Streaming SP11

This post will provide a hands on example of configuring, running and testing the Streaming Web Service (“SWS”) for SAP HANA Smart Data Streaming. It will explain how to set various settings in HANA cockpit, and how to test the Streaming Web Service using the Chrome Advanced REST Client.

Detailed documentation for the Streaming Web Service can be found at: Help Portal, Streaming Web Service

Prerequisites

• Access to a HANA system with the Smart Data Streaming SP11 option installed and configured.
• HANA user ID with permissions to create, compile, deploy and run a Smart Data Streaming project. This can be the HANA SYSTEM user which is granted full permissions for streaming as part of the install process.
• Ability to connect to the Linux host running the Smart Data Streaming as the <sid>adm user

What is the Streaming Web Service

The Streaming Web Service (SWS) is one of the SAP HANA Smart Data Streaming (“SDS”) adapters. SWS is a scalable gateway which provides HTTP-based access to SAP HANA SDS. It provides greater scalability and performance than the older Web Service Provider (“WSP”). Like the WSP, the SWS can receive requests from multiple web services and publish data to multiple projects.  This contrasts with other adapters which only send and receive data from one source and publish to one project.

The SWS has 2 significant functional differences from the WSP:

1. 1) The REST and Websocket request and message formats supported by the SWS are different from the message formats previously required by the WSP, which means that upgrading WSP applications to work with SWS will require changes to the JSON message body.
2. 2) SWS uses a token based authentication in contrast to the WSP user ID & password authentication. In order to publish to a stream or window using SWS, a client must first obtain an authorization token by sending a REST request with its credentials and requested permissions. The Streaming Web Service responds with a token which must be included in the Authorization header of subsequent requests. The token expires after a specified duration which can be set in the configuration. When this occurs, the client must send a new authorization request.

Configuration

The configuration details for the SWS are stored in the HANA database, and can be accessed through the HANA cockpit.

To edit these settings, log into HANA cockpit and select the Streaming Cluster Configuration tile.

Navigate to the Streaming Web Service tab, and you will see several properties that can be changed. If unchanged, their default value will be used. Full documentation can be found here: Help Portal, Configuring the Streaming Web Service.

Leave the Host blank. This means it will be able to bind to any interface server. Also note down the Port number, which should be 9093 by default. We will be using 9093 for this tutorial. Next, ensure that SSL Enabled is unchecked for this tutorial. For convenience, check the Auto Start Enabled box. This will allow the SWS to always start up with the SDS cluster. Finally, under Token Validity (seconds) change the value to 3000.

If you would like to turn on tracing, you can by changing Trace Enabled and/or Trace HTTP Enabled to “true”. You can check outputs in HANA Studio by looking at the log and trace files under your host’s diagnosis files. This tab is located in your HANA system’s administration window, which can be accessed by double clicking the system while in the Administration Console perspective. Look for the prefix “streamingserver~”

Going back to HANA cockpit, leave the default values for the other entries. Once you are done, click save.

Running the Service

Since we checked the Auto Start Enabled box, SWS will automatically start up with the SDS cluster. However, for the setting changes to take effect, we will need to restart the SDS cluster. On the SDS node, run HDB stop and HDB start. When the SDS cluster starts up again, SWS will be running.

Alternatively, you can manually turn SWS on and off using the streamingclusteradmin utility, located in $STREAMING_HOME/bin on your SDS node.

Run this command in $STREAMING_HOME/bin to start SWS, after filling in the values:

./streamingclusteradmin -uri esps://<host>:<port> --auth=user-password --username=<username> --password=<password> --start_sws

Run this command in $STREAMING_HOME/bin to start SWS, after filling in the values:

./streamingclusteradmin -uri esps://<host>:<port> --auth=user-password --username=<username> --password=<password> --stop_sws

Part B: Testing Streaming Web Service connectivity using the Chrome “Advanced Rest Client”

This section will walk throughthe basic message formats of Streaming Web Service, using the Google Chrome App “Advanced Rest Client”.

You can download the Advanced Rest Client for the Chrome browser here: https://chrome.google.com/webstore/detail/advanced-rest-client/hgmloofddffdnphfgcellkdfbfbjeloo

Prerequisite

For the purpose of this tutorial, make sure the Ssl Enabled setting is unchecked. We will be using HTTP, not HTTPS to test the SWS.

Sample Streaming Project

Provided below is a sample streaming project consisting of a single input stream. We will use the MACHINEDATA input stream to receive JSON messages via a REST post.

CREATE INPUT STREAM MACHINEDATA SCHEMA (

       MACHINEID string ,

       EVENT_TIME msdate ,

       EVENT_NAME string ,

       EVENT_DESCRIPTION string ,

       EVENT_VALUE string ) ;

Note: This input stream is taken from the Freezer Monitoring tutorial. If you have already completed the Freezer Monitoring tutorial, you can use the existing Freezer Monitoring project for this tutorial.

Configuring the Streaming Project to Enable Web Services

Start by navigating to the “SAP HANA Streaming Development” perspective and opening the project configuration (.ccr) file for your streaming project. Under the “Advanced” tab in the Project Configurations window that appears, click on “Project Deployment (Non-HA)”. Enable Web Services by checking the corresponding box in the Project Deployment Details and setting the value in the drop down menu to “true”.

Note: If the check box to the left of the “Web Service Enabled” option is not checked, then the option value of true or false is ignored.

Start the Sample Project

Once you have configured the sample project to enable web services, compile and run the project on the SDS server. In the SAP HANA Streaming Run-Test perspective, open the stream view for the MACHINEDATA stream. This is where we will see the event records appear after we post them from the REST client.

Advanced Rest Client

When first opened, the Advanced Rest Client app will look like this:

IMPORTANT: Click the drop down arrow (circled in blue), and delete the auto generated “Key | Value” query parameter line(s). This is done by clicking the x. We need to do this or else the URL will be wrong.

There are a few relevant fields on this interface which we will be using, and they are labeled above as HOST, PATH, HTTP Methods, Header, and Payload.

Sending and Receiving Data

First, let’s test if SWS is running, by obtaining the version string. Type/change the following in the corresponding fields, and leave all other fields blank:

HOST:

http://< my.streaming.host.com>:9093

PATH:

/1/version

HTTP Method:
GET

The format for HOST is http://< my.streaming.host.com>:<port>. The <my.streaming.host.com> is the fully qualified domain name (“FQDN”) or TCP/IP address of the SDS node of the HANA system. Make sure the port number matches the one in the streamingwebservice.xml file which should be 9093 if you kept the default port number as recommended above.

Click send. Scroll to the bottom of the screen, and you should be able to see the returned string inside the Response field. The format is:

     Response: { "version-string":"<version-string>" }

Obtaining the Authentication Token

Before you can publish messages, SWS requires that you request for a token for the specific resource you intend to access. The returned token then needs to be included in the headers of subsequent POSTs.

The Payload of the JSON message can be used to request access to multiple resources at once, and the format is the following:

[

{ "privilege":"view|write", "resourceType":"project|stream", "resource":"<workspace>/ <project>/<stream>"},

{ "privilege":"view|write", "resourceType":"project|stream", "resource":"<workspace>/ <project>/<stream>"},

……………….

]

Note: The token is only valid for a set number of seconds, which is defined in HANA cockpit. The default value is 300, but we changed it to 3000 for the sake of this tutorial. After this time period, a new token will need to be requested.

For this test, we need to request a token that will allow us to publish to the MACHINEDATA stream of the freezer_monitoring project running under the default workspace. Type/change the following in the corresponding fields, and leave all other fields blank:

HOST:

http://<my.streaming.host.com>:9093

PATH:

/1/authorization

HTTP Method:

POST

Header:

Authorization: Basic <auth-data>

Payload:

[

{ "privilege":"write", "resourceType":"stream", "resource":"default/freezer_monitoring/MACHINEDATA"}

]

Fill <auth-data> with the base64-encoded authorization data for the SYSTEM user. This can be generated automatically by the Chrome App, just toggle from the Raw tab to the Form tab, put “Authorization” in the key. Then click on the value field, and click the pencil icon.

Enter the username SYSTEM and password in the subsequent screen, and it will fill in the value field for you.

This is what it looks like:

Click send, and the response should contain the token. The format is:

     Response: [ { "sws-token":"<token>" } ]

Copy the token onto your clipboard for the next step.

Publishing to a Stream

The data that will be written to the MACHINEDATA input stream in the SDS project needs to be defined as a JSON payload. The JSON message body can be sent in either of 2 formats:

1. A “map” format
2. An “array” format

The structure of the “map” format is:

[

{ "ESP_OPS":"<opcode>", "<field1>":<value1>, "<field2>":<value2>, ... },

{ "ESP_OPS":"<opcode>", "<field1>":<value1>, "<field2>":<value2>, ... }

...

]

Each line sends values to the corresponding fields, based on field name. Any unspecified fields will be sent the value of NULL.

The structure of the “array” format is:

[

[ "<opcode>", <value1>, <value2>, <value3>, ...],

[ "<opcode>", <value1>, <value2>, <value3>, ...]

...

]

With this method, every field must be sent a value, and the order they are typed must correspond to the schema.

For this tutorial we will use the map format. Enter the following into the corresponding fields of the Advanced Rest Client, and leave all other fields blank:

HOST:

http://dewdflhana2370.emea.global.corp.sap:9093

PATH:

/1/workspaces/default/projects/freezer_monitoring/streams/MACHINEDATA

HTTP Method:

POST

Header:

Authorization: SWS-Token "sws-token"="<token>"

Payload:

[

{"ESP_OPS":"i", "MACHINEID":"Adv Rest Cli", "EVENT_TIME": "2012-01-23T04:45:36.123", "EVENT_NAME":"DOOR", "EVENT_DESCRIPTION":"sws", "EVENT_VALUE":"DoorOpen"}

]

Paste the token you obtained previously in the place of <token>. Your screen should look like this:

Click send. If successful, you will see this:

You can now go to HANA Studio to check for the output:

Expired Tokens

If your token expired during this process, you will receive the error “419 Authentication Timeout”.

If this occurs, you will need to go back to the step Obtaining the Authentication Token, obtain a new token, and retry the step Publishing to a Stream.