The SAP Internet of Things (IoT) services introduce the concept of 'things,' tangible entities designed for efficient data acquisition and exchange. A 'thing' encompasses physical objects, regardless of size or complexity, that holds significance from a user or application standpoint. Companies, acting as business partners, manufacture and provide services for these 'things.'
In an IoT scenario, properties undergo frequent changes, and if these changes occur every second or every hour, configuring property set types like TimeSeriesData or DerivedData becomes essential. This categorization is crucial for representing and managing data such as temperature, humidity, or pressure.
In the presented scenario, numerous 'things' generate substantial data, some of which is stored in the Coldstore. To process and extract the entire data batch efficiently, a comprehensive process was created. SAP Integration Suite manages the file generation and process flow, while SAP Data Intelligence handles the extensive file processing power. SAP HANA Cloud is used for storing the data securely.
Picture 1 - Architecture Overview
The technical setup involves the SAP IoT Coldstore, which triggers file exports through REST calls, SAP Integration Suite for orchestrating the process, and SAP Data Intelligence for robust file processing. SAP HANA Cloud serves as the ultimate destination for storing the processed data.
To initiate file exports in SAP IoT Coldstore, three REST calls are made to initiate data export, read the status of data export, and download the exported data. The data is stored in a zip archive containing multiple gzip archives, each containing a CSV with time series data.
Initiating the data export - https://coldstore-export-sap.cfapps.eu10.hana.ondemand.com/v1/InitiateDataExport/<Alert-Type>?timera...>
This will return a RequestId.
Reading the status of the data export - https://coldstore-export-sap.cfapps.eu10.hana.ondemand.com/v1/DataExportStatus?requestId=<RequestId>
When the file is ready to be downloaded, the response will look like this :
{
"RequestId": "2E44E36AD1B2482481C71E3C3BE2B276",
"Status": "The file is available for download."
}
Downloading the data export - https://coldstore-downloader-sap.cfapps.eu10.hana.ondemand.com/v1/DownloadData('BF192FDB37BF47B5A3C0...')
Below is a sample with csv data :
"_TIME","I_C2103FB0E2444D229E8FB4B45D426244","I_693CC407CDE449719DFC0E6DD99CE643","I_2203C0BE13194426A8B6DBC3F1C1F64F","I_0FD1C08A690A4822BD5154A104347BDD","I_3E9F3D3016BB4832A2E65DFF9DEEFEFB","I_7CA1A47DB03B474EB0DBBFE95D024164","I_A3DCE207F9384AA2B4BB11E6E29B4147","I_8FA84523D2084AE39ACEE8704686863B","I_7E51BB58470C4F39A86CEB6779D4D932","I_99CC3B7559CB4641A0835951D425D025","I_546C0BFE572647AB89D2A1E537D9179A","I_97921DB6DFD2406C866609C11362AA23","I_BFED37C910624E60BDA9642668C577C8","I_A7634ECFF3DE4DC2A3C2049CCA404E60","I_22BF3697253244459ED7E694C015C931","I_CC32F7C157494C40880B0316148DF06D","I_ED0EF7F0E55B4CFB8F5A7487E88DCAF9","I_99CC3B7559CB4641A0835951D425D025","I_EA1FF2EBC99845CAB2C39F3A7AE04D82","I_D840BACA0AF145189D4527E374AA295B","I_97921DB6DFD2406C866609C11362AA23","I_7DA27CA1219F44BD954F7691C7CCAE39","I_546C0BFE572647AB89D2A1E537D9179A","I_B89480BF97464303B136A31795D846D4","I_12DE2BEB1DE14403A761077299EBEED6","I_AA6550D7FC214B5AB8A2C3D91397F753","I_BFED37C910624E60BDA9642668C577C8","I_8FA84523D2084AE39ACEE8704686863B","I_97921DB6DFD2406C866609C11362AA23","I_A7634ECFF3DE4DC2A3C2049CCA404E60","I_99CC3B7559CB4641A0835951D425D025","I_2203C0BE13194426A8B6DBC3F1C1F64F","I_7CA1A47DB03B474EB0DBBFE95D024164","I_0ADEB1031D404206B04254D0759ADECC","equipmentId","modelId","indicatorGroupId","templateId"
"1686002400000","","","","","","","","","","","0.0","0.0","0.0","0.0","0.0","0.0","0.0","0.0","0.0","0.0","0.0","0.0","0.0","0.0","0.0","0.0","1.0","","4CDF29D259004DF6B627D767F5627B46","IG_4C5BC3CE776E4CA8AA1075CC25005195","9721581EB74E4EE185184589EDC45961","E072E4D388F24D07AED12D7F3DA16214"
"1685998800000","","","","","","","","","","","0.0","0.0","0.0","0.0","0.0","0.0","0.0","0.0","0.0","0.0","0.0","0.0","0.0","0.0","0.0","0.0","1.0","","4CDF29D259004DF6B627D767F5627B46","IG_4C5BC3CE776E4CA8AA1075CC25005195","9721581EB74E4EE185184589EDC45961","E072E4D388F24D07AED12D7F3DA16214"
A bit harder to interpret than a standard CSV but later on we’ll see how we can achieve this.
SAP Integration Suite is instrumental in controlling the entire process of file generation and handling the flow. Utilizing JMS Queuing, the suite checks for the readiness of downloaded files, which is crucial due to the time-consuming generation of time series data for a large batch of 'things.'
After the file is fully processed and downloaded by SAP Integration Suite, we use the following Flow Steps to handle the archive types :
Zip Splitter with a .gz file pattern that extracts and splits in multiple messages the .gzips found inside the main .zip.
GZIP Decompression that extracts the csv data inside the .gzip.
Picture 2 - iFlow Artifact that sends the data to SAP Data Lake
After the csvs are ready, we send the data to SAP Data Lake - https://<host>-hana.ondemand.com/webhdfs/v1/shared/<folder>/${header.ThingId}_${property.Split_Number}_${property.Timestamp}?op=CREATE&data=true
After placing the csv files to SAP Data Lake, we insert to JMS a message containing the URL to the file. That message is being picked up by another iFlow artifact that triggers in SAP Data Intelligence the file import from SAP Data Lake towards the SAP HANA Cloud database. If everything goes as planned, the same message as before ( containing the body ) is put back to JMS.
This final message is being picked up by a flow that triggers the file deletion after they were processed by SAP Data Intelligence.
Picture 3 - iFlow Artifact that triggers the delete file operation
For the Storage Gateway Delete Operation I used the /webhdfs/v1/{path}?op=DELETE API that is well documented here - https://api.sap.com/api/storagegateway/path/delete .
If the deletion was completed, the process ends.
Further, I will explain what was configured & developed for mass processing of the CSVs and the part of sending actual data to the SAP HANA Cloud database.
Picture 4 - SAP Data Intelligence Graph
By using the OpenAPI Servlow operator, we have exposed a REST inbound point for the graph.
From SAP Integration Suite we receive the file path we use that to read the file from SAP Data Lake. The CSV data is being done through the python script which simply reads the data, extracts the equipment ids, the alert timestamps and their values and creates another CSV readable by the SAP HANA Cloud Database.
The final javascript script helps us send the proper answer back to SAP Integration Suite. It has 3 inputs, and in case any of the calls when trying to read, process or send the csv fails, it will send back to SAP Integration Suite as '{ "result" : "error"}', otherwise it will send '{ "result" : "ok"}'. This response is being used to keep/remove the message from the JMS Queue.
In SAP HANA Cloud, the data is persisted in a simple table with columns such as ID, EQUIPMENT_ID, ALERT_ID, RECEIVEDAT, and VALUE. This table captures essential information about the 'things' and their associated data.
ID | EQUIPMENT_ID | ALERT_ID | RECEIVEDAT | VALUE |
1a2b3c4d-5e6f-7g8h-9i0j1k2l3m4n | X12A34B567C890D1E23F456G78H90IJ1 | ABCDEF1234567890GHIJKLMNOPQRSTUVWXYZ | 1/23/2024 8:15 | 1.2 |
3x2gc3522f9f-02caxf32-r3ca-y7b4 | X98Y76Z54W32V21U09T87S65R43Q21P | 9876543210ZYXWVUTSRQPONMLKJIHGFEDCBA | 1/23/2024 10:30 | 2.5 |
9i0j1k2l3m4n-1a2b3c4d-5e6f7g8h | X34D56E78F90G12H34I56J78K90L12M | WXYZABCDEFGHIJKLMNOPQRSTUV9876543210 | 1/23/2024 12:45 | 0.9 |
4o5p6q7r8s9t1u2v3w4x5y6z7a8b9c0 | X56H78J90K12L34M56N78P90Q12R34S | ZYXWVUTSRQPONMLKJIHGFEDCBA9876543210 | 1/23/2024 15:00 | 3.1 |
1b2a3r4-5e6r7-8i9o0u1s2-3d4a5t6a7b8l9e0 | X78C90D12E34F56G78H90I12J34K56L | ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789 | 1/23/2024 17:15 | 1.8 |
3s2o3m4e-02caxf32-r3ca-y7b4 | X21N34O56P78Q90R12S34T56U78V90W | ZYXWVUTSRQPONMLKJIHGFEDCBA9876543210 | 1/23/2024 19:30 | 2.3 |
9l0i1n2e3-1a2b3c4d-5e6f7g8h | X90I12J34K56L78M90N12O34P56Q78R | 9876543210ZYXWVUTSRQPONMLKJIHGFEDCBA | 1/23/2024 21:45 | 0.6 |
4g5c6b7b8a9l1l2e3s4-5f6-7i8l9t0e | X12U34V56W78X90Y12Z34A56B78C90D | ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789 | 1/24/2024 0:00 | 2 |
1z2x3c4v5b6-5n6m7b8-9i0l1k2j3 | X45E67F89G01H23I45J67K89L01M23N | ZYXWVUTSRQPONMLKJIHGFEDCBA9876543210 | 1/24/2024 2:15 | 1.5 |
3x2gc3522f9f-02caxf32-r3ca-y7b4 | X89O12P34Q56R78S90T12U34V56W78X | ABCDEF1234567890GHIJKLMNOPQRSTUVWXYZ | 1/24/2024 4:30 | 0.4 |
Connecting SAP IoT Coldstore data with SAP HANA Cloud through SAP Data Intelligence and Integration Suite has proven to be a powerful and seamless process. The orchestration capabilities of SAP Integration Suite ensure a smooth flow of data, and SAP Data Intelligence's processing power handles extensive file operations efficiently. The integration suite simplifies complex workflows, making it easy to connect and manage large amounts of data. The robust architecture seamlessly blends different SAP components, demonstrating the strength of SAP's end-to-end solutions for IoT data management and analytics.
For any questions or suggestions, feel free to post here or in the special Q&A Section.
Thank you!
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
User | Count |
---|---|
12 | |
6 | |
5 | |
4 | |
3 | |
3 | |
3 | |
3 | |
3 | |
3 |