- SAP Managed Tags:
Updated March 8, 2018
In previous posts, I have discussed multiple aspects of Internet of Things security, in particular about the edge infrastructure of machines, devices, sensors, gateways, etc. and how to apply good and established principles and practices from Enterprise IT security, where we can.
What we haven't looked into before, is how this ties in with SAP software products. The recommendations we made have been fairly generic, and are widely applicable. These recommendations are absolutely still valid, both in an SAP context and beyond, and remain good sources that outline SAP's general approach to IoT Security. These previous articles - the series, as well as others - focus on protecting the edge and how to get data securely from the edge into a cloud environment, including the SAP Cloud Platform. This is a key building block in securing Industrial IoT solutions.
However, we can always get more concrete and practical. By looking at an entire solution landscape with actual products, we understand better how everything fits together. And what better candidate is there than to look at one of the most valuable and at the same time most security-critical landscapes: Digital Manufacturing - the combination of business systems, the Operation Technology worlds of SCADA and ICS, AI/ML, sensors and Cloud.
Industrial IoT is often described as bridging the IT-OT divide, or IT-OT Convergence. The security implications of this are severe and I have written more about the security of OT networks here. SAP has a long history in manufacturing, and you can read more about SAP's vision on Digital Manufacturing in this document. It seemed therefore a natural step to outline this landscape from a security perspective.
Below you can see a diagram of the Digital Manufacturing landscape and its various data flows:
Since this is a rather complex picture, here is a version of that same diagram highlighting the function of the various components in the stack.
At the very bottom, we have the OT network of machines, PLCs, and other OT specific components, as well as a sensor network, providing additional monitoring of this environment (temperature, humidity, pressure, vibration, etc., etc.). This is connected with the IT environment via an IT-OT Gateway with edge processing capabilities, that (primarily) exchanges date with SAP's Manufacturing Suite (specifically MII), which itself interacts with the ERP system.
At the top is the SAP Cloud Platform, hosting the SAP Manufacturing Cloud, consisting of a number of Cloud-based Manufacturing applications, like Digital Manufacturing Insights (DMI), and Predictive Quality Management (PQM) as well as future manufacturing applications. They receive their data securely via the Cloud Connector, essentially a trusted VPN tunnel, that connects a customer's IT environment at a plant or office, with the SAP Cloud Platform. These applications can also leverage and integrate with other cloud applications like Asset Intelligent Network (AIN), Predictive Maintenance (PdMS) and the Big Data Services (BDS) and ML/AI capabilities of the platform.
This was a very quick run-through, but perhaps a pattern is already emerging: we have here a number of distinct networks, with clearly identifiable points where networks are crossed. I have earlier discussed the Internet of Things as a network of isolated networks and this is essentially a concrete implementation of that. Let's overlay the diagram one more time, this time to show the separate networks involved:
The OT Network only communicates with the IT Network via the IT-OT Gateway, which has two network cards, one facing into the OT layer, the other into the IT network. Connectivity is through the PlantConnectivity (PCo) component, ideally over OPC-UA which provides encryption and mutual trust between components using X.509 PKI certificates. While PCo supports a variety of industrial protocols, many of those are not particularly secure, and OPC-UA is essentially designed (from a security perspective) to provide a TLS-like encrypted tunnel with mutual signing to ensure that each component in the communication can be assured of the other's identity. This practically means that PCo will know it is talking to the correct PLC, and the PLC knows it is talking to the correct PCo instance. OPC-UA is SAP's preferred method to connect into the OT layer. We understand that OPC-UA is not yet universally adopted, but we encourage to put it on your roadmap, if it is not already. This is an important element in securing and shielding the OT Network from outside harm.
PCo sends the data it receives from the machine layer to the MII Data Services layer over HTTPS. Alternatively, it can be sent to another PCo instance. This can be useful architecturally to further separate the OT environment from IT systems, depending on the network topology (for instance, by running PCo in the OT network itself, passing data to a PCo instance in the IT network), or how the solution is deployed across multiple manufacturing plants. In addition, the data can be sent to a HANA database first, and retrieved by MII from there. This provides further options.
On the same gateway, we have an instance of Edge Services, which can be used to batch and collect data, and apply calculations over it (for instance, averaging values over a time period), or apply business rules to data. Edge Services comes in two flavors, an on premise version (formerly known as Dynamic Edge Processing) and a cloud version. In both scenarios, it shares its results back with PCo, which itself shares the data with other system components, and can push data to a local data lake. Both traffic flows, even while on the same host, are encrypted. The cloud version relies on the IoT Services Gateway Edge (dashed box) which communicates with the IoT Services in the cloud, where it is backed by the Edge Services Policy Server. This allows central life cycle management of Edge Services and the sharing of data for longer term analysis within the SAP Cloud Platform, where it can be used by other applications, as well as machine learning services.
The integration of MII with ERP, or other SAP components like EWM, additional MII instances or PI, runs typically over either a JRA or JCo connection for RFC calls, protected and encrypted with SNC. This is completely standard and is almost certainly already in place.
To push data into the cloud from this environment, we use Smart Data Integration (SDI) through the Cloud Connector. The Cloud Connector is a VPN proxy that creates a secure encrypted tunnel into the cloud environment that makes the remote database appear local to on-premise IT systems, and includes an access list that determines which hosts in the IT network can communicate with this proxy. In addition to this, of course, the applications sending data through this channel also need to be valid database users on the cloud data storage.
Note that in this landscape, there is no direct link between the machine layer and the cloud. That's not to say that such connections might not exist. Many OEM manufacturers of these machines themselves connect them to their own machine/device clouds, often through data diodes. While SAP certainly sees uses for data diodes - which allow traffic movement only in one direction - we prefer this approach, where first on-premise machine data is enriched by business context, and only then, from within the IT network, moved to the cloud.
The machines in the OT layer, especially newer ones, have a lot of sensors already built-in. However, this may not always be sufficient for the sort of monitoring customers want to do. An interesting approach to adding further sensor data that can be added to the business context and machine data already retrieved via PCo is the use of a separate sensor network altogether.
This is exactly what a large German chemical manufacturer did. They were happy with the security of their OT environment, but felt the machines didn't give them all the data they were looking for. The question then was how to do this. One option was to add sensors directly into the OT network itself, but there was a fear that this would increase risk in the environment, and rightly so. The other option was to run a separate sensor network over LPWAN.
I have written before about micro controller sensors operating over LPWAN, and the encryption scheme SAP developed for exactly those types of devices and networks. In brief, these are low-powered sensors, often running on battery power, and making their own RF link connection to an LPWAN base station wirelessly, be it often at severe restrictions on bandwidth and number of messages per hour or day. From the base station the data flows through to the LPWAN network provider cloud, from where it is transported to the desired destination - i.e. SAP Cloud Platform in our scenario in cloud-to-cloud integration. What makes this interesting in this context is that the sensors themselves are physically located and placed on- or near the machines they are meant to monitor, while at the same time being completely isolated from the OT network. They are in a separate network of their own that has no ability to communicate with either the OT network or the IT network, which is why in the diagram it is separated out completely with the green border. An equivalent would be your smart phone: when not connected to any WiFi networks, you are still connected to the internet, but through a GSM network.
One issue is that while the communication from the base station to the LPWAN provider cloud, and from the provider cloud to another cloud environment like SAP Cloud Platform is easy to secure (IPSEC, TLS), the link between the device itself and the base station is weak - exactly where we need it to be strong. Breaking into cloud-to-cloud connection is not a trivial exercise, but there is no strong protection exactly where it is most vulnerable to attack: near the LPWAN devices themselves. For this reason, SAP developed an encryption scheme for low-power devices, so we can encrypt the data on the device itself, and only resolve that upon receipt of the message in the SAP Cloud Platform. The combination of low-power encryption for these LPWAN sensor with encrypted and signed traffic over OPC-UA in the OT layer, creates a well isolated and secure machine environment, where the data from the LPWAN sensors is not merged and combined with the business context and machine data from the IT and OT networks until all these data streams reach the Cloud environment.
Manufacturing customers, I suspect, already will recognize a lot of components in this landscape. The ECC-MII-ME with PCo stack has been around for some time, and in many cases will already be in place at your manufacturing plants. This is good. We already know how to protect that, and I am always a big fan of security approaches that build upon existing practices. It's always easier to do something well you are already familiar with, rather than get it right the first time on something new. TLS and SNC protections, as well as appropriate user roles, should not be new to SAP system operators in these locations.
The functionality and reach into the machine layer has certainly increased, though, since PCo now also can send messages back down into the machine layer. This provides the order- or ERP-driven manufacturing capabilities, and therefore risk is increased. Which is exactly why we are so strong on the use of OPC-UA.
Moreover, connecting the cloud applications only from the SAP IT systems through the Cloud Connector, provides a secure way of getting machine data, with the business context, into that environment in a way that is low-impact, while providing insight over multiple plant locations - something that in the past was not very easy to do.
The only truly new element in the landscape is the LPWAN sensor network, and this is clearly additional in this environment to provide further information and insight into the manufacturing process and their environmental variables. Should you have no immediate need for additional temperature, pressure, vibration or other sensor data, because the OT equipment is already giving you what you currently need, then by all means leave it out of your landscape.
Continuity is good. Securing things we know is always easier than security things we don't. Rather than major disruption to get onto the path of IoT-assisted digital manufacturing, we chose continuity, and building on top of an established core. Much of the configuration of SAP systems doesn't change, whereas many customers are already using the Cloud Connector in Production.
This continuity extends in the heavy use of digital certificates and encrypted traffic in the landscape. This is familiar at least to security professionals, and even if OPC-UA is new to them, they will very quickly pick up on it, as it leverage technologies that are familiar. We have all been setting up TLS and SNC for years. We know about good password policies, and much of the additional infrastructure to protect the environments - firewalls, VPN solutions and data diodes, IPS/IDS systems, NAC/SDN, and SIEM tools and security analytics - is either taken directly from enterprise IT security, or have equivalents in the more unusual networks and protocols (for instance, monitoring solutions for OT environments, that are aware of industrial protocols).
I completely understand reluctance to adopt IIoT or Digital Manufacturing for security concerns. The risk without question goes up. But I would argue this risk is controllable. We can make smart choices that limit the attack vector, and avoid bad decisions that jeopardize the security of networks that we are currently reasonably comfortable with. We believe that we took a workable approach, and from a security perspective, I am rather satisfied with this landscape. But that is not to say that it cannot be improved, or that there aren't additional things we can do. Since we're all on a path towards figuring this out - unfortunately, the vast majority of public content on IoT Security still focuses on the risks and dangers, rather than provide solutions - I would love to hear your reactions to this, and even more so objections and suggested improvements. I would invite you therefore to use the comments section below to continue the conversation.
Digital Manufacturing White Paper
Plant Connectivity Product Page
Edge Services Product Page
In previous posts, I have discussed multiple aspects of Internet of Things security, in particular about the edge infrastructure of machines, devices, sensors, gateways, etc. and how to apply good and established principles and practices from Enterprise IT security, where we can.
What we haven't looked into before, is how this ties in with SAP software products. The recommendations we made have been fairly generic, and are widely applicable. These recommendations are absolutely still valid, both in an SAP context and beyond, and remain good sources that outline SAP's general approach to IoT Security. These previous articles - the series, as well as others - focus on protecting the edge and how to get data securely from the edge into a cloud environment, including the SAP Cloud Platform. This is a key building block in securing Industrial IoT solutions.
However, we can always get more concrete and practical. By looking at an entire solution landscape with actual products, we understand better how everything fits together. And what better candidate is there than to look at one of the most valuable and at the same time most security-critical landscapes: Digital Manufacturing - the combination of business systems, the Operation Technology worlds of SCADA and ICS, AI/ML, sensors and Cloud.
Industrial IoT is often described as bridging the IT-OT divide, or IT-OT Convergence. The security implications of this are severe and I have written more about the security of OT networks here. SAP has a long history in manufacturing, and you can read more about SAP's vision on Digital Manufacturing in this document. It seemed therefore a natural step to outline this landscape from a security perspective.
Digital Manufacturing Landscape
Below you can see a diagram of the Digital Manufacturing landscape and its various data flows:
Since this is a rather complex picture, here is a version of that same diagram highlighting the function of the various components in the stack.
High-level functional description
At the very bottom, we have the OT network of machines, PLCs, and other OT specific components, as well as a sensor network, providing additional monitoring of this environment (temperature, humidity, pressure, vibration, etc., etc.). This is connected with the IT environment via an IT-OT Gateway with edge processing capabilities, that (primarily) exchanges date with SAP's Manufacturing Suite (specifically MII), which itself interacts with the ERP system.
At the top is the SAP Cloud Platform, hosting the SAP Manufacturing Cloud, consisting of a number of Cloud-based Manufacturing applications, like Digital Manufacturing Insights (DMI), and Predictive Quality Management (PQM) as well as future manufacturing applications. They receive their data securely via the Cloud Connector, essentially a trusted VPN tunnel, that connects a customer's IT environment at a plant or office, with the SAP Cloud Platform. These applications can also leverage and integrate with other cloud applications like Asset Intelligent Network (AIN), Predictive Maintenance (PdMS) and the Big Data Services (BDS) and ML/AI capabilities of the platform.
A network of networks
This was a very quick run-through, but perhaps a pattern is already emerging: we have here a number of distinct networks, with clearly identifiable points where networks are crossed. I have earlier discussed the Internet of Things as a network of isolated networks and this is essentially a concrete implementation of that. Let's overlay the diagram one more time, this time to show the separate networks involved:
The OT Network only communicates with the IT Network via the IT-OT Gateway, which has two network cards, one facing into the OT layer, the other into the IT network. Connectivity is through the PlantConnectivity (PCo) component, ideally over OPC-UA which provides encryption and mutual trust between components using X.509 PKI certificates. While PCo supports a variety of industrial protocols, many of those are not particularly secure, and OPC-UA is essentially designed (from a security perspective) to provide a TLS-like encrypted tunnel with mutual signing to ensure that each component in the communication can be assured of the other's identity. This practically means that PCo will know it is talking to the correct PLC, and the PLC knows it is talking to the correct PCo instance. OPC-UA is SAP's preferred method to connect into the OT layer. We understand that OPC-UA is not yet universally adopted, but we encourage to put it on your roadmap, if it is not already. This is an important element in securing and shielding the OT Network from outside harm.
PCo sends the data it receives from the machine layer to the MII Data Services layer over HTTPS. Alternatively, it can be sent to another PCo instance. This can be useful architecturally to further separate the OT environment from IT systems, depending on the network topology (for instance, by running PCo in the OT network itself, passing data to a PCo instance in the IT network), or how the solution is deployed across multiple manufacturing plants. In addition, the data can be sent to a HANA database first, and retrieved by MII from there. This provides further options.
On the same gateway, we have an instance of Edge Services, which can be used to batch and collect data, and apply calculations over it (for instance, averaging values over a time period), or apply business rules to data. Edge Services comes in two flavors, an on premise version (formerly known as Dynamic Edge Processing) and a cloud version. In both scenarios, it shares its results back with PCo, which itself shares the data with other system components, and can push data to a local data lake. Both traffic flows, even while on the same host, are encrypted. The cloud version relies on the IoT Services Gateway Edge (dashed box) which communicates with the IoT Services in the cloud, where it is backed by the Edge Services Policy Server. This allows central life cycle management of Edge Services and the sharing of data for longer term analysis within the SAP Cloud Platform, where it can be used by other applications, as well as machine learning services.
The integration of MII with ERP, or other SAP components like EWM, additional MII instances or PI, runs typically over either a JRA or JCo connection for RFC calls, protected and encrypted with SNC. This is completely standard and is almost certainly already in place.
To push data into the cloud from this environment, we use Smart Data Integration (SDI) through the Cloud Connector. The Cloud Connector is a VPN proxy that creates a secure encrypted tunnel into the cloud environment that makes the remote database appear local to on-premise IT systems, and includes an access list that determines which hosts in the IT network can communicate with this proxy. In addition to this, of course, the applications sending data through this channel also need to be valid database users on the cloud data storage.
Note that in this landscape, there is no direct link between the machine layer and the cloud. That's not to say that such connections might not exist. Many OEM manufacturers of these machines themselves connect them to their own machine/device clouds, often through data diodes. While SAP certainly sees uses for data diodes - which allow traffic movement only in one direction - we prefer this approach, where first on-premise machine data is enriched by business context, and only then, from within the IT network, moved to the cloud.
LPWAN sensor network
The machines in the OT layer, especially newer ones, have a lot of sensors already built-in. However, this may not always be sufficient for the sort of monitoring customers want to do. An interesting approach to adding further sensor data that can be added to the business context and machine data already retrieved via PCo is the use of a separate sensor network altogether.
This is exactly what a large German chemical manufacturer did. They were happy with the security of their OT environment, but felt the machines didn't give them all the data they were looking for. The question then was how to do this. One option was to add sensors directly into the OT network itself, but there was a fear that this would increase risk in the environment, and rightly so. The other option was to run a separate sensor network over LPWAN.
I have written before about micro controller sensors operating over LPWAN, and the encryption scheme SAP developed for exactly those types of devices and networks. In brief, these are low-powered sensors, often running on battery power, and making their own RF link connection to an LPWAN base station wirelessly, be it often at severe restrictions on bandwidth and number of messages per hour or day. From the base station the data flows through to the LPWAN network provider cloud, from where it is transported to the desired destination - i.e. SAP Cloud Platform in our scenario in cloud-to-cloud integration. What makes this interesting in this context is that the sensors themselves are physically located and placed on- or near the machines they are meant to monitor, while at the same time being completely isolated from the OT network. They are in a separate network of their own that has no ability to communicate with either the OT network or the IT network, which is why in the diagram it is separated out completely with the green border. An equivalent would be your smart phone: when not connected to any WiFi networks, you are still connected to the internet, but through a GSM network.
One issue is that while the communication from the base station to the LPWAN provider cloud, and from the provider cloud to another cloud environment like SAP Cloud Platform is easy to secure (IPSEC, TLS), the link between the device itself and the base station is weak - exactly where we need it to be strong. Breaking into cloud-to-cloud connection is not a trivial exercise, but there is no strong protection exactly where it is most vulnerable to attack: near the LPWAN devices themselves. For this reason, SAP developed an encryption scheme for low-power devices, so we can encrypt the data on the device itself, and only resolve that upon receipt of the message in the SAP Cloud Platform. The combination of low-power encryption for these LPWAN sensor with encrypted and signed traffic over OPC-UA in the OT layer, creates a well isolated and secure machine environment, where the data from the LPWAN sensors is not merged and combined with the business context and machine data from the IT and OT networks until all these data streams reach the Cloud environment.
Continuity: building on an established core
Manufacturing customers, I suspect, already will recognize a lot of components in this landscape. The ECC-MII-ME with PCo stack has been around for some time, and in many cases will already be in place at your manufacturing plants. This is good. We already know how to protect that, and I am always a big fan of security approaches that build upon existing practices. It's always easier to do something well you are already familiar with, rather than get it right the first time on something new. TLS and SNC protections, as well as appropriate user roles, should not be new to SAP system operators in these locations.
The functionality and reach into the machine layer has certainly increased, though, since PCo now also can send messages back down into the machine layer. This provides the order- or ERP-driven manufacturing capabilities, and therefore risk is increased. Which is exactly why we are so strong on the use of OPC-UA.
Moreover, connecting the cloud applications only from the SAP IT systems through the Cloud Connector, provides a secure way of getting machine data, with the business context, into that environment in a way that is low-impact, while providing insight over multiple plant locations - something that in the past was not very easy to do.
The only truly new element in the landscape is the LPWAN sensor network, and this is clearly additional in this environment to provide further information and insight into the manufacturing process and their environmental variables. Should you have no immediate need for additional temperature, pressure, vibration or other sensor data, because the OT equipment is already giving you what you currently need, then by all means leave it out of your landscape.
Continuity is good. Securing things we know is always easier than security things we don't. Rather than major disruption to get onto the path of IoT-assisted digital manufacturing, we chose continuity, and building on top of an established core. Much of the configuration of SAP systems doesn't change, whereas many customers are already using the Cloud Connector in Production.
This continuity extends in the heavy use of digital certificates and encrypted traffic in the landscape. This is familiar at least to security professionals, and even if OPC-UA is new to them, they will very quickly pick up on it, as it leverage technologies that are familiar. We have all been setting up TLS and SNC for years. We know about good password policies, and much of the additional infrastructure to protect the environments - firewalls, VPN solutions and data diodes, IPS/IDS systems, NAC/SDN, and SIEM tools and security analytics - is either taken directly from enterprise IT security, or have equivalents in the more unusual networks and protocols (for instance, monitoring solutions for OT environments, that are aware of industrial protocols).
I completely understand reluctance to adopt IIoT or Digital Manufacturing for security concerns. The risk without question goes up. But I would argue this risk is controllable. We can make smart choices that limit the attack vector, and avoid bad decisions that jeopardize the security of networks that we are currently reasonably comfortable with. We believe that we took a workable approach, and from a security perspective, I am rather satisfied with this landscape. But that is not to say that it cannot be improved, or that there aren't additional things we can do. Since we're all on a path towards figuring this out - unfortunately, the vast majority of public content on IoT Security still focuses on the risks and dangers, rather than provide solutions - I would love to hear your reactions to this, and even more so objections and suggested improvements. I would invite you therefore to use the comments section below to continue the conversation.
Digital Manufacturing White Paper
Plant Connectivity Product Page
Edge Services Product Page
8 Comments
