During my days as SAP employee I was responsible for various data integration solutions around the ERP system, today I am propagating its next evolution.
Recap
ERP systems aim to be a digital twin of the reality. When a product is shipped to a customer in the physical world, the ERP's shipping document represents the data of this action. In the pursuit towards excellence, users of the ERP system have the natural tendency to add more detailed information, e.g. by customizations to store the additional data.
The next logical step is to integrate external information as well, for example the shipping status. No rocket science to call the shipping agent's API and to retrieve the delivery status. But by adding more data, the complexity and hence the total costs of ownership of the ERP system increases exponentially.
Singularity
To be frank, this does not scale. The various systems and modules should rather look and feel like one system. A tightly coupled choreography working in harmony - the singularity.
Integration of processes is too rigid for that. If all works like clockwork the system is fine. But if reality throws a spanner into the works, the predefined processes fall apart leaving a cascade of failures in its wake.
We should rather change the way we perceive ERPs:
- A chain of integrated processes → Network
- Reality represented as data structures → software makes decisions which impact the physical world
- Managing the business in tacts → Realtime business
- Simple if-this-than-that business logic → considering a lot of related information
Looking at these changes, many SAP themes can be found, e.g. "Hana is the platform for realtime business", "Automation", "Predictive", etc. In the area of integration customers can do better.
API first? Customer first!
The phrase
API first means that there is a well defined interface others can use. If there is an API for everything, anything is possible. This assumption, catchy as it is, has multiple obvious issues.
- How many APIs does the SAP API Business Hub expose for S/4Hana? As of today it is 290, a fraction of the functions an ECC system has. The obvious interfaces exist, like creating a sales order, but it is far from complete.
- Every API is rather heavy weight, which has pros and cons. On the plus side it is powerful and many aspects of e.g. a sales order can be changed, but a single field update like setting a status flag, has lots of internal overhead.
- When is an API perfectly well defined? If it provides all the required fields, including customer specific and caller specific data. Or putting it differently: Never.
Things get even more nasty when APIs depend on each other. A Sales order application must know the list of current materials, the user picks some, for each the available quantity must be queried, production orders are created and much more. The advantage of the ECC System is that all is done in one database, and coupling these deeply nested processes is well supported by database transactions. With external systems things become much more complicated and, in consequence, unreliable, thus expensive.
see
microservice dependency hell
The ONE project to harmonize the data models makes the transformations between the various data structures easy - there are no transformations necessary if all systems use the same structures. It is an expensive option but as these costs are covered by SAP, no complaints here. It would not be required for the proposed solution but helps in those cases where the ONE data model works perfectly.
From a state machine...
The key reason of the integration drama is how the ERP systems was intended to be used: As a black box. Somebody performs an action:
create sales order. The ERP system sets multiple database changes in motion and when completed, returns. Only a few methods exist (and are used even less frequently) that allow external actions on internal state changes.
In a world where the central ERP is just that, the core system to govern the business, there is no need for such things. But these times are long gone. The rules are no longer just predefined static workflows. The ERP is one of many systems, the most important one, but still just one of many that need to collaborate.
Collaboration means communication.
...to a team player
Let me make a suggestion:
- Instead of providing many different APIs for everything and consumers must call each, let the ERP system communicate all state changes to the outside world proactively instead. People can listen to that stream or an interesting subsets, but it is the consumer's choice.
- Instead of providing attributes the developer of the API deemed to be useful, it should communicate everything.
- Instead of informing that there was a change, provide the change.
This would look and feel for other consumers like a social media stream: "Hey, customerX just bought 100 pieces of productY!". For humans this stream would be overwhelming but consuming it with software services, maybe even combining data from multiple streams, does enable powerful applications with little effort.
Feasibility and costs
The requirement to stay informed at all times is a direct effect of today's society and technology determinism. Both pointing to the same outcome - communication networks. In other words, it is expected by customers and the technology to enable that has been developed - in the Big Data world.
For Big Data, the volumes of changes created per second in even the largest ERP systems is laughable compared to what companies like Netflix, LinkedIn an others produce in the same time interval. The requirements have a slightly different emphasis, e.g. consistency of data gets more important but can be fulfilled.
At the end, using an ERP system as described as a communication hub is not only more powerful but also cheaper to implement than even a single process integration step.
Technically speaking, the solution does capture all changes of all S/4Hana data structures, move them into Apache Kafka and is available for everybody interested and with the proper permissions. This provides the outgoing communication channel where our coworker, the ERP system, keeps everybody informed about every single state change it made.
In Kafka we have all options to implement more business logic without impacting the ERP system at a fraction of the costs.
The result is a win-win-win. The ERP system is used for bookings, one single API - the Kafka Data Producer - keeps all partner systems informed in realtime. I love it when a plan comes together.