Technology Blogs by Members
Explore a vibrant mix of technical expertise, industry insights, and tech buzz in member blogs covering SAP products, technology, and events. Get in the mix!
cancel
Showing results for 
Search instead for 
Did you mean: 
MarianVatafu
Explorer
527

What is a ‘Thing’ ?

 

The SAP Internet of Things (IoT) services introduce the concept of 'things,' tangible entities designed for efficient data acquisition and exchange. A 'thing' encompasses physical objects, regardless of size or complexity, that holds significance from a user or application standpoint. Companies, acting as business partners, manufacture and provide services for these 'things.'

 

Time Series Data for Things

 

In an IoT scenario, properties undergo frequent changes, and if these changes occur every second or every hour, configuring property set types like TimeSeriesData or DerivedData becomes essential. This categorization is crucial for representing and managing data such as temperature, humidity, or pressure.

 

Scenario Overview

 

In the presented scenario, numerous 'things' generate substantial data, some of which is stored in the Coldstore. To process and extract the entire data batch efficiently, a comprehensive process was created. SAP Integration Suite manages the file generation and process flow, while SAP Data Intelligence handles the extensive file processing power. SAP HANA Cloud is used for storing the data securely.

 

Architecture Overview

 

diagram.png

 

 

 

 Picture 1 - Architecture Overview

 

Technical Setup

 

The technical setup involves the SAP IoT Coldstore, which triggers file exports through REST calls, SAP Integration Suite for orchestrating the process, and SAP Data Intelligence for robust file processing. SAP HANA Cloud serves as the ultimate destination for storing the processed data.

 

SAP IoT Coldstore - Triggering File Export

To initiate file exports in SAP IoT Coldstore, three REST calls are made to initiate data export, read the status of data export, and download the exported data. The data is stored in a zip archive containing multiple gzip archives, each containing a CSV with time series data.

Initiating the data export - https://coldstore-export-sap.cfapps.eu10.hana.ondemand.com/v1/InitiateDataExport/<Alert-Type>?timera...>

This will return a RequestId.

Reading the status of the data export - https://coldstore-export-sap.cfapps.eu10.hana.ondemand.com/v1/DataExportStatus?requestId=<RequestId>

When the file is ready to be downloaded, the response will look like this :

 

 

 

{

    "RequestId": "2E44E36AD1B2482481C71E3C3BE2B276",

    "Status": "The file is available for download."

}

 

 

 

Downloading the data export - https://coldstore-downloader-sap.cfapps.eu10.hana.ondemand.com/v1/DownloadData('BF192FDB37BF47B5A3C0...')

Below is a sample with csv data :

 

 

 

"_TIME","I_C2103FB0E2444D229E8FB4B45D426244","I_693CC407CDE449719DFC0E6DD99CE643","I_2203C0BE13194426A8B6DBC3F1C1F64F","I_0FD1C08A690A4822BD5154A104347BDD","I_3E9F3D3016BB4832A2E65DFF9DEEFEFB","I_7CA1A47DB03B474EB0DBBFE95D024164","I_A3DCE207F9384AA2B4BB11E6E29B4147","I_8FA84523D2084AE39ACEE8704686863B","I_7E51BB58470C4F39A86CEB6779D4D932","I_99CC3B7559CB4641A0835951D425D025","I_546C0BFE572647AB89D2A1E537D9179A","I_97921DB6DFD2406C866609C11362AA23","I_BFED37C910624E60BDA9642668C577C8","I_A7634ECFF3DE4DC2A3C2049CCA404E60","I_22BF3697253244459ED7E694C015C931","I_CC32F7C157494C40880B0316148DF06D","I_ED0EF7F0E55B4CFB8F5A7487E88DCAF9","I_99CC3B7559CB4641A0835951D425D025","I_EA1FF2EBC99845CAB2C39F3A7AE04D82","I_D840BACA0AF145189D4527E374AA295B","I_97921DB6DFD2406C866609C11362AA23","I_7DA27CA1219F44BD954F7691C7CCAE39","I_546C0BFE572647AB89D2A1E537D9179A","I_B89480BF97464303B136A31795D846D4","I_12DE2BEB1DE14403A761077299EBEED6","I_AA6550D7FC214B5AB8A2C3D91397F753","I_BFED37C910624E60BDA9642668C577C8","I_8FA84523D2084AE39ACEE8704686863B","I_97921DB6DFD2406C866609C11362AA23","I_A7634ECFF3DE4DC2A3C2049CCA404E60","I_99CC3B7559CB4641A0835951D425D025","I_2203C0BE13194426A8B6DBC3F1C1F64F","I_7CA1A47DB03B474EB0DBBFE95D024164","I_0ADEB1031D404206B04254D0759ADECC","equipmentId","modelId","indicatorGroupId","templateId"

"1686002400000","","","","","","","","","","","0.0","0.0","0.0","0.0","0.0","0.0","0.0","0.0","0.0","0.0","0.0","0.0","0.0","0.0","0.0","0.0","1.0","","4CDF29D259004DF6B627D767F5627B46","IG_4C5BC3CE776E4CA8AA1075CC25005195","9721581EB74E4EE185184589EDC45961","E072E4D388F24D07AED12D7F3DA16214"

"1685998800000","","","","","","","","","","","0.0","0.0","0.0","0.0","0.0","0.0","0.0","0.0","0.0","0.0","0.0","0.0","0.0","0.0","0.0","0.0","1.0","","4CDF29D259004DF6B627D767F5627B46","IG_4C5BC3CE776E4CA8AA1075CC25005195","9721581EB74E4EE185184589EDC45961","E072E4D388F24D07AED12D7F3DA16214"

 

 

 

A bit harder to interpret than a standard CSV but later on we’ll see how we can achieve this.

SAP Integration Suite - Orchestrating the Process

SAP Integration Suite is instrumental in controlling the entire process of file generation and handling the flow. Utilizing JMS Queuing, the suite checks for the readiness of downloaded files, which is crucial due to the time-consuming generation of time series data for a large batch of 'things.'

After the file is fully processed and downloaded by SAP Integration Suite, we use the following Flow Steps to handle the archive types :

Zip Splitter with a .gz file pattern that extracts and splits in multiple messages the .gzips found inside the main .zip.

GZIP Decompression that extracts the csv data inside the .gzip.

SEND DATA TO DI.png

 

 

 

 Picture 2 - iFlow Artifact that sends the data to SAP Data Lake

After the csvs are ready, we send the data to SAP Data Lake - https://<host>-hana.ondemand.com/webhdfs/v1/shared/<folder>/${header.ThingId}_${property.Split_Number}_${property.Timestamp}?op=CREATE&data=true

After placing the csv files to SAP Data Lake, we insert to JMS a message containing the URL to the file. That message is being picked up by another iFlow artifact that triggers in SAP Data Intelligence the file import from SAP Data Lake towards the SAP HANA Cloud database. If everything goes as planned, the same message as before ( containing the body ) is put back to JMS.

This final message is being picked up by a flow that triggers the file deletion after they were processed by SAP Data Intelligence.

delete data from di.png

 

 Picture 3 - iFlow Artifact that triggers the delete file operation

 

For the Storage Gateway Delete Operation I used the /webhdfs/v1/{path}?op=DELETE API that is well documented here - https://api.sap.com/api/storagegateway/path/delete .

If the deletion was completed, the process ends.

Further, I will explain what was configured & developed for mass processing of the CSVs and the part of sending actual data to the SAP HANA Cloud database.

SAP Data Intelligence - Mass Processing of CSVs

DI_FLOW.png

 

Picture 4 - SAP Data Intelligence Graph

 

By using the OpenAPI Servlow operator, we have exposed a REST inbound point for the graph.

From SAP Integration Suite we receive the file path we use that to read the file from SAP Data Lake. The CSV data is being done through the python script which simply reads the data, extracts the equipment ids, the alert timestamps and their values and creates another CSV readable by the SAP HANA Cloud Database.

The final javascript script helps us send the proper answer back to SAP Integration Suite. It has 3 inputs, and in case any of the calls when trying to read, process or send the csv fails, it will send back to SAP Integration Suite as '{ "result" : "error"}', otherwise it will send '{ "result" : "ok"}'. This response is being used to keep/remove the message from the JMS Queue.

SAP HANA Cloud

In SAP HANA Cloud, the data is persisted in a simple table with columns such as ID, EQUIPMENT_ID, ALERT_ID, RECEIVEDAT, and VALUE. This table captures essential information about the 'things' and their associated data.

ID

EQUIPMENT_ID

ALERT_ID

RECEIVEDAT

VALUE

1a2b3c4d-5e6f-7g8h-9i0j1k2l3m4n

X12A34B567C890D1E23F456G78H90IJ1

ABCDEF1234567890GHIJKLMNOPQRSTUVWXYZ

1/23/2024 8:15

1.2

3x2gc3522f9f-02caxf32-r3ca-y7b4

X98Y76Z54W32V21U09T87S65R43Q21P

9876543210ZYXWVUTSRQPONMLKJIHGFEDCBA

1/23/2024 10:30

2.5

9i0j1k2l3m4n-1a2b3c4d-5e6f7g8h

X34D56E78F90G12H34I56J78K90L12M

WXYZABCDEFGHIJKLMNOPQRSTUV9876543210

1/23/2024 12:45

0.9

4o5p6q7r8s9t1u2v3w4x5y6z7a8b9c0

X56H78J90K12L34M56N78P90Q12R34S

ZYXWVUTSRQPONMLKJIHGFEDCBA9876543210

1/23/2024 15:00

3.1

1b2a3r4-5e6r7-8i9o0u1s2-3d4a5t6a7b8l9e0

X78C90D12E34F56G78H90I12J34K56L

ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789

1/23/2024 17:15

1.8

3s2o3m4e-02caxf32-r3ca-y7b4

X21N34O56P78Q90R12S34T56U78V90W

ZYXWVUTSRQPONMLKJIHGFEDCBA9876543210

1/23/2024 19:30

2.3

9l0i1n2e3-1a2b3c4d-5e6f7g8h

X90I12J34K56L78M90N12O34P56Q78R

9876543210ZYXWVUTSRQPONMLKJIHGFEDCBA

1/23/2024 21:45

0.6

4g5c6b7b8a9l1l2e3s4-5f6-7i8l9t0e

X12U34V56W78X90Y12Z34A56B78C90D

ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789

1/24/2024 0:00

2

1z2x3c4v5b6-5n6m7b8-9i0l1k2j3

X45E67F89G01H23I45J67K89L01M23N

ZYXWVUTSRQPONMLKJIHGFEDCBA9876543210

1/24/2024 2:15

1.5

3x2gc3522f9f-02caxf32-r3ca-y7b4

X89O12P34Q56R78S90T12U34V56W78X

ABCDEF1234567890GHIJKLMNOPQRSTUVWXYZ

1/24/2024 4:30

0.4

 

Conclusions

 

Connecting SAP IoT Coldstore data with SAP HANA Cloud through SAP Data Intelligence and Integration Suite has proven to be a powerful and seamless process. The orchestration capabilities of SAP Integration Suite ensure a smooth flow of data, and SAP Data Intelligence's processing power handles extensive file operations efficiently. The integration suite simplifies complex workflows, making it easy to connect and manage large amounts of data. The robust architecture seamlessly blends different SAP components, demonstrating the strength of SAP's end-to-end solutions for IoT data management and analytics.

For any questions or suggestions, feel free to post here or in the special Q&A Section.

Thank you!

Labels in this area