Question Re: Forward IoT data to smart data streaming in Technology Q&A

Forward IoT data to smart data streaming

Former Member — Sat, 03 Feb 2024 21:56:40 GMT

hello - i'm setting up a scenario as below.

1. sensor sends data to iot services on HCP

2. configured service to forward sensor data to smart data streaming on cal.sap.com

3. Registered service from cal.sap.com to HCP - as mentioned in http://help-legacy.sap.com/saphelp_hana_options_sds_adapt/helpdata/en/68/df4e6a1cbd43f1890ce58e012f4f7d/content.htm

4. Created project/workspace/etc and they are working fine

Issue - the sensor data doesn't seem flow to smart data streaming service.. not much useful info in the log file. Any suggestions.

Secondly - where do i see the data forwarded from iot service in cal (i configured HANA studio, but there is no data. i'm using SYSYEM schema) - any suggestion on which table holds the data.

thanks for your help

pandari

Re: Forward IoT data to smart data streaming

anton_levin — Fri, 19 May 2017 05:18:39 GMT

Hello Pandari, in spite you say the logs do not show something useful, we would still need your SAP Cloud Platform account ID and landscape you are working on to check them. You may either share them by replying on this answer or via direct message in case you do not want to share those publicly (I will send you an initial message to start the conversation). Regards, Anton

Re: Forward IoT data to smart data streaming

Former Member — Fri, 19 May 2017 11:42:28 GMT

Hello Pandari,

have you used the SDS studio plugin for Eclipse to create the project/workspace/etc.? If you change to the "SAP HANA Streaming Run Test" perspective, you can connect to your SDS, select your running input stream and open the stream viewer window. There you can see all incoming data. The same perspective also provide the possibility to test your stream with sample data.

Whether the data is stored somewhere persistent in a table depends on your stream configuration project. If you use there a HANA output adapter for your input stream, you can specify the "landing table", where the data should be send to.

These topics above are related to SDS. If you have further questions there, please have a look in the official SAP SDS documentation

https://help.sap.com/viewer/product/HANA_STREAMING/2.0.01/en-US

Regarding your problems that data is not forwarded from SAP Cloud Platform Internet of Things Service to SDS, we need some more information. Have you created an SDS Processing Service in your MMS instance? The procedure doing that is described here: https://help.sap.com/viewer/7436c3125dd5491f939689f18954b1e9/Cloud/en-US/b81fd20c92574587ae745aa692f2976c.html

The SDS project structure must match to your used message type as described in the "SDS project setup" section.

Kind regards,

Ralf

Re: Forward IoT data to smart data streaming

Former Member — Fri, 19 May 2017 13:44:49 GMT

Thanks Anton - I will send you the account details directly.

Thanks Ralf for your comments. Response to your comments:

1. have you used the SDS studio plugin for Eclipse to create the project/workspace/etc.? - Yes, and i can see the test data.i'm able to connect to the server, etc.

2. Whether the data is stored somewhere persistent in a table depends on your stream configuration project. - Thanks for the clarification. i thought by default the streaming data is stored in a table or somewhere.

3. Have you created an SDS Processing Service in your MMS instance? - yes. i created the structure including the g_ fields. All the field names in the iot service is reflected in the streaming structure

4. I will send you the log data.

Thanks

Pandari

Re: Forward IoT data to smart data streaming

Former Member — Fri, 19 May 2017 23:14:31 GMT

Attached are the trace files.

Thx

Pandari

Re: Forward IoT data to smart data streaming

anton_levin — Sat, 20 May 2017 16:57:48 GMT

Hello Pandari, our monitoring tool and the logs show that the iotmms instance on your account is not operating normally. It lacks the connectivity to the underlying DB. Since "processing service" configurations are stored in the DB, this might be a potential issue why iotmms is unable to forward data to SDS. What type of the DB are you using? Is it a shared HANA or you have acquired the MDC instance? In case of MDC ensure that it is up and running (be aware that is stops automatically after some period of time). You might need to re-configure SDS processing service after you fix the DB binding issue. Regards, Anton

Re: Forward IoT data to smart data streaming

Former Member — Sat, 20 May 2017 18:26:54 GMT

Hello Anton - thanks. did you get a chance to review the log files attached. i'm using MDC instance, and i made sure it's up and running.

thx

pandari

Re: Forward IoT data to smart data streaming

anton_levin — Sat, 20 May 2017 19:22:06 GMT

Yes, i looked into them (also remotely) but according to the metrics, iotmms is not able to connect to the DB. If you say MDC is up and running, then check if data source binding to iotmms is there (maybe it needs to be re-binded because of the password change etc). I would also recommend to re-strart iotmms process and as suggested above, check the SDS configurations.

Re: Forward IoT data to smart data streaming

Former Member — Mon, 22 May 2017 21:30:51 GMT

hello Anton - I'm getting the error below. This is in the iotmms default trace file.

2017 05 22 21:23:47#+00#ERROR#com.sap.iotservices.mms.processing.sds.connection.StreamingWebServiceHttpsConnection##anonymous#Delayed request processing for device '6115856b-7fec-4b94-b930-09704ecff342'#na#pXXXtrial#iotmms#web##SWS Token could not be retrieved. Response Code: 405|

Additional info:

FILE_TYPE:DAAA96DE-B0FB-4c6e-AF7B-A445F5BF9BE2 FILE_ID:1495477078755 ENCODING:[UTF8|NWCJS:ASCII] RECORD_SEPARATOR:124 COLUMN_SEPARATOR:35 ESC_CHARACTER:92

HEADER_END 2017 05 22 18:18:07#+00#ERROR#com.sap.iotservices.mms.processing.ProcessingServiceRegistry##anonymous#localhost-startStop-1#na#p296796trial#iotmms#web##Could not retrieve the service mappings from the data base.java.io.IOException: Failed to find a corresponding processing service named 'sds' for device type '3041ca0ef3a86a9c736e' and message type 'e1b0068faba1203f6661'.

21:23:47#+00#ERROR#com.sap.iotservices.mms.processing.sds.connection.StreamingWebServiceHttpsConnection##anonymous#Delayed request processing for device '6115856b-7fec-4b94-b930-09704ecff342'#na#p296796trial#iotmms#web##SWS Token could not be retrieved. Response Code: 405|

Any thoughts on how to resolve this.

Thx

Pandari

Re: Forward IoT data to smart data streaming

Former Member — Mon, 22 May 2017 22:11:33 GMT

further update. i deleted all the previous registrations and registered the smart data streaming service, with "wss" protocol, then i got the message below in "iotmms"

2017 05 22 22:06:10#+00#ERROR#com.sap.iotservices.mms.processing.sds.connection.StreamingWebServiceWssConnection##anonymous#IoT MMS pool thread-2#na#pXXXtrial#iotmms#web##No websocket connection available.|

Re: Forward IoT data to smart data streaming

Former Member — Mon, 22 May 2017 22:13:23 GMT

Registed to iotmms using: https://iotmmspXXXXtrial.hanatrial.ondemand.com/com.sap.iotservices.mms/v1/api/http/processing

{ "deviceType":"3041ca0ef3a86a9c736e", "messageType":"e1b0068faba1203f6661", "processingServices":[ { "name":"sds", "properties":{ "mms.processing.sds.protocol":"wss", "mms.processing.sds.host":"52.166.240.XXX", "mms.processing.sds.port":30026, "mms.processing.sds.user":"SYSTEM", "mms.processing.sds.pwd":"XXXX", "mms.processing.sds.workspace":"default", "mms.processing.sds.project":"zstreamproj", "mms.processing.sds.input_stream":"ziotstream", "mms.processing.sds.proxy_enabled":"false" } } ] }

Re: Forward IoT data to smart data streaming

anton_levin — Tue, 23 May 2017 06:35:42 GMT

Dear Pandari, I still see that your iotmms instance is not able to connect to the underlying DB. Please, as stated above in my comment from May, 20th - bring iotmms to the operating state first. If it is not feasible with MDC instance, then I would suggest to switch back to shared HANA schema. Without underlying DB it is not possible for iotmms to operate with its processing services, since all configurations are stored there. Regards, Anton

Re: Forward IoT data to smart data streaming

Former Member — Thu, 25 May 2017 00:35:26 GMT

i reconfigured to use shared instance, now i'm getting the error below, which is similar to what i received before. Any other thoughts.

Re: Forward IoT data to smart data streaming

Former Member — Thu, 25 May 2017 00:55:44 GMT

i restarted the iot services, and deleted the previous registration, and registered the service as wss and https, still it failed.

+00#ERROR#com.sap.iotservices.mms.processing.sds.connection.StreamingWebServiceWssConnection##anonymous#IoT MMS pool thread-3#na#p296796trial#iotmms#web##No websocket connection available.| 2017 05 25 00:51:46#+00#ERROR#com.sap.iotservices.mms.processing.sds.connection.StreamingWebServiceWssConnection##anonymous#IoT MMS pool thread-3#na#p296796trial#iotmms#web##No websocket connection available.| 2017 05 25 00:52:03#+00#ERROR#com.sap.iotservices.mms.processing.sds.connection.StreamingWebServiceWssConnection##anonymous#IoT MMS pool thread-3#na#p296796trial#iotmms#web##No websocket connection available.| 2017 05 25 00:52:14#+00#ERROR#com.sap.iotservices.mms.processing.sds.connection.StreamingWebServiceWssConnection##anonymous#IoT MMS pool thread-3#na#p296796trial#iotmms#web##No websocket connection available.|

Re: Forward IoT data to smart data streaming

anton_levin — Thu, 25 May 2017 05:13:53 GMT

Hello Pandari, now monitoring tool indicates that the iotmms operates normally, thanks. But in the logs I see the following: "Failed to find a corresponding processing service named 'sds' for device type '304***36e' and message type 'e1b***661'". May you please post here a screenshot of your "T_IOT_PROCESSING_SERVICE_CONFIG" table? IoT MMS Cockpit -> Application Data (tile, top-left corner) -> T_IOT_PROCESSING_SERVICE_CONFIG table. (Please, mask user sensitive info if required). Regards, Anton

Re: Forward IoT data to smart data streaming

Former Member — Thu, 25 May 2017 20:27:20 GMT

Please see attached. iot-processing-service.jpg

Re: Forward IoT data to smart data streaming

Former Member — Mon, 29 May 2017 08:57:18 GMT

Hello Pandari,

it seems that your SDS is not configured properly. To ensure that, could you please send a POST request with an HTTP client (e.g. POSTMAN) to your SDS authorization service as described here in the SDS guide: https://apps.support.sap.com/sap/support/knowledge/preview/en/2344255

Method: POST

URL:

https://<system>sws<account>.<landscape-suffix>:443/1/authorization

Header:

- Content-Type: application/json

- Authorization: BASIC (with your username and password to access Smart Data Streaming)

Body:

[ { "privilege":"write", "resourceType":"stream", "resource":"<your workspace>/<your project>/<your stream>" } ]

as far as I can see this from your configuration picture, the resource string should be "default/zstreamproj/ziotstream"

If the response code is ...

- 200 with the response [{"sws-token":"<token>"}], your SDS service seems to be configured correctly and we should investigate the interaction between IoT Service MMS and SDS in detail

- 401, your credentials are not valid

- 405, your SDS is not configured correctly. Maybe you have not enable the Streaming Webservice (SWS) or Web Services Provider (WSP) functionality - Than please follow the configuration instructions in the link provided above.

Kind regards,

Ralf

Re: Forward IoT data to smart data streaming

Former Member — Tue, 30 May 2017 23:30:31 GMT

hello ralf - i'm using SAP cloud appliance (cal.sap.com) + azure, and i have the ip address of SDS instance. so i constructed my url as https://<ip-address>:443//1/authorization, content json, basic authentication - user id/pass, and raw payload - [ { "privilege":"write", "resourceType":"stream", "resource":"default/zstreamproj/ziotstream" } ]

I got - "The requested url can't be reached". I'm able to access this url from eclipse and using browser.

i changed the port from 443 to 30026, i got "install self signed certificate" error..

any thoughts on how the url should look like.

thx

pandari

Re: Forward IoT data to smart data streaming

RobertWaywell — Wed, 31 May 2017 14:52:17 GMT

If you are trying to publish to a CAL instance, then the host name and port will be different than if you are working in HCP. For example, HCP handles the routing from a generic port 443 to the actual SWS port. The default port for websocket connections on the SWS itself is 9093. When trying to connect to the SWS on an Azure VM the address for the uri should be "https://<external or public ip address of the Azure host>:9093/..."

Re: Forward IoT data to smart data streaming

Former Member — Thu, 01 Jun 2017 13:37:06 GMT

Pandari, were you able to get the authorization token from the SDS in your CAL instance with the link Robert provided?
You can also check the version with "https://<external or public ip address of the Azure host>:9093/1/version". Do you get a response there?