How to build an Integration Architecture for the Intelligent Enterprise

[stevang]

How do we build an Integration Architecture for the Intelligent Enterprise?

Okay, let’s start with the usual disclaimers – we need to be pragmatic and use common sense (as always in the Architecture area). I have presented some views on Lean EA (Enterprise Architecture) in my article Enterprise Architecture in the era of Agile…

Common sense, and no “worshiping” of any EA frameworks, fine – but we could still use some guidelines, right?

This article is providing a high-level overview – how to build Integration Architecture in the Organization. As usually, I am providing Solution Concept (an idea or working idea), not a Solution itself.

Integration Guiding Principles

On the Enterprise Architecture level, I would define 2 main guiding principles:

1. Consumer driven integration – we build for business needs; APIs should not dictate the integration;


2. Events driven integration – no need to replicate and create multiple copies of the same “data” in the different Systems;

On the Solution Architecture level (when solutioning & designing integrations), I would define 2+1 main guiding principles:

1. Use standard –whenever possible;


2. Reuse existing – as much as possible;


3. Keep it well documented (this is "+1" as this is something we should commonly enforce in all domains);

The Approach (or Methodology)

We do need to put some order in the Integration Architecture. And let us not reinvent the “hot water”…

I would start with SAP Integration Solution Advisory Methodology (ISA-M)[1]. No, it is not used only for SAP Eco space. ISA-M defines (among other things) Integration Domains, Integration Styles and Use Cases Patterns in a very agnostic way.

In this article, I will stay with the agnostic part of ISA-M, as I firmly believe Enterprise Architecture should not be vendor driven (at least on the high-level), just as I believe Integration Architecture should not be API driven[2].

ISA-M proposes a very useful definition and split into main Integration Styles and accompanying Use Cases Patterns:

• Process Integration Style; and Master Data Integration, A2A, B2B associated Use Case Pattern;


• Data Integration Style; and ETL, Data Orchestration associated Use Case Pattern;


• User Integration Style; and UI Integration, Chatbot Integration associated Use Case Pattern


• Etc,

Use Case Patters can have many Integration Services

E.g. Customer (Business Object) Integration Service within Process Integration Style (as part of the business process of creating/updating of the Customer/Account) could be part of Master Data Integration Use Case Pattern; but Customer (Business Object) Integration Service within Data Integration Style (for DWH and reporting) could be part of ETL Use Case Pattern…

Sales Order (Business Object) Integration Service within Process Integration Style (as business process of Order Fulfillment) could be part of A2A Use Case Pattern…

Service Ticker (Business Object) within User Integration Style (as iFrame) could be part of UI Integration Use Case Pattern…

While each Organization has it’s one Integration Services, some common “major” services are usually built around key Business Object (or Data Object). Here as well, fortunately, there are many proposed “Best Practice” definitions. I find LeanIX[3][4] approach very useful for building Integration Architecture (and Enterprise Architecture overall).

Now, let’s clearly understand one Business Object does not represent one Integration Service.

E.g. Business Object Sales Order is essentially part of several Integration Services like Sales Order Replicate, Sales Order Create/Update etc.

Also, one Integration Service may have more than one Interface – and this is not because of GET, POST, PUT etc. – this would still be one interface, but with different methods. Let’s take an example – we might be designing Product Replicate Integration Service from S/4HANA which consists of two interfaces – one is MATMAS IDoc for Material, and one is CLFMAS IDoc for (Material) Classification.

And, of course, each System may have more Integration Services, and versa-vise one Integration Services may be used to connect more than two Systems.

A bit more on SAP ISA-M…

Of course, when building SAP specific Integration Architecture one can further benefit with various accelerators. For more details on ISA-M and how to start using it, pls visit the official page[5]. To learn more about current template versions (version 4 is the latest available, pls. check article SAP Integration Solution Advisory Methodology: Template version 4.0 available now), or get some general overview in articles (e.g. article Integration Solution Advisory Methodology (ISA-M): Define Integration Guidelines for Your Organizati...).

For the actual realization of the Integration Architecture (in projects), if we have e.g. S/4HANA as our Core ERP System, and if we have SAP CPI as a Middleware, we could & should try to benefit from numerous accelerators available in SAP API Business Hub. Remember, we keep it simple – if we have standard solutions provided by SAP, connecting  SAP/non-SAP Systems (and SAP API Business Hub provides accelerators for many non/SAP Systems), let’s just use it. Amazingly enough, people are still getting surprised how many “modern” OData Services (or Microservices) can be used out-of-the-box with S/4HANA… For a long time, S/4HANA is much, much more than IDocs and RFCs…

Integration Services – Characteristics

In order to have proper governance, it is a good idea to further define Integration Services per its specific Characteristics. Although, there is no rulebook, I would see following main Characteristics of each Integration Service:

• Integration Patterns:


• Integration Function:


• Data Integration Scope


• Execution Style

Integration Patterns

For the high-level overview, we can define Integration Patterns using comprehensive (but very specific  list provided by SAP API Business Hub[5], or extensive list and guides (agnostic but technology driven) provided by Enterprise Integration Patterns[6], or go for a more pragmatic (very condensed & "high-level") list from MuleSoft[7], or find an inspiration somewhere else. I’ll normally go for a combination (fitting specific needs), but would usually start with a more pragmatic approach (and develop "deep-dive" later).

Data Migration Integration Pattern – there is one Source System of the Business Object; one-time bulk data is PUSH/PULL from the Source System to the Target System(s).

Broadcast – there is one Source System of the Business Object; data is PUSH from the Source System to the Target System(s). Usually this is an Event Notification or Data Replication.

One-way synchronization – there is one Source System of the Business Object; data is PUSH/PULL from the Source System to the Target System(s); where Source System (or the Initiating System, in general) is notified of the synchronization status.

Two-way synchronization – both (or many) Systems are equal Source & Target Systems of the Business Object; data is PUSH/PULL from one System to the other System(s); where each System (or the Initiating System, in general) is notified of the synchronization status.

Data Correlation – both (or many) Systems are natural & independent Source Systems of the Business Object; intersection of data which is created naturally & independently is PUSH/PULL from one System to the other System(s); where each System (or the Initiating System, in general) is notified of the synchronization status. E.g. two Sales Agents creates the same Account in the different Systems – those Accounts will be synchronized, but if only one Sales Agent creates Account in one System – this Account will not be synchronized to the other System (will not be visible to the other Sales Agent);

Data Aggregation – there can be many Source Systems but only one Target System of the Business Object; data is PUSH/PULL from the Source System(s) to the Target System. E.g. pulling off the  IoT readings; or ETL into DWH.

*) There could be also “complex” Integration Patterns as a combination of several Integration Patterns.

Integration Function

The list is straight-forward, and self-explanatory.

Create – new data record(s) is PUSH from the Source System to the Target System(s). E.g. calling POST method in the remote Service.

Read – data record(s) is PULL from the Source System to the Target System(s). E.g. calling GET method in the remote Service, referencing to the specific record(s).

Update – update existing record(s) is PUSH (Request to change record) from the Source System to the Target System(s). E.g. calling PATCH method in the remote Service, referencing to the specific record(s).

Delete – delete existing record(s) is PUSH (Request to delete record) from the Source System to the Target System(s). E.g. calling DELETE method in the remote Service, referencing to the specific record(s).

*) There could be also “complex” Integration Functions (e.g. MERGE) as a combination of several Integration Functions.

Data Integration Scope

Here as well, list is pretty-much straight-forward, and self-explanatory.

Full dataset object – all data instances of the Business Object (e.g. full update); usually one time (for Data Migration) or on periodical batch job (e.g. for Data Correlation etc.);

Partial dataset – range of the Business Object (e.g. delta or filtered per specific criteria); usually on periodical batch job (e.g. periodical update); or on-event (e.g. calling GET method to read data from the remote System);

Single record – one (or only few) instance of the Business Object; usually on-event trigger (e.g. new record, or changed record etc.);

*) In all cases, integration could cover only a subset of the Business Object’s attributes.

Integration Execution

Integration can be executed in the synchronous way (=Sync) or in asynchronous way (=Async).

Sync is a single thread, so only one operation will run at a time – System sending Request will wait for that Request to be answered by the other System (=Response), before continuing with other operations. Usually, this is used to encapsulate the transaction(s).

Async is multi-thread, so System can run in parallel other operations – System sending Request Message will not wait for any Response, before continuing with other operations. The other System may still send back (at some point of time) separate Response Massage, e.g. to update the status of the processing etc. Although the exchange of Request Message and Response Message may be very fast in many cases (almost near-real time), there is no “guarantee”, no “time-out”, and System sending Request Message will not “block” its operations waiting for the Response Message.

Final Thoughts

As said, all this is just a Solution Concept, just an idea – there is no strict rulebook how to build “sustainable” Integration Architecture - an Architect responsible for building the Strategy in the respective domain, will be building the Integration Architecture in his/her Organization. And please note; "one size, does not fit all" (Organizations)…

I am inviting you to keep following relevant blogs and community resources, post and answer questions, and read other posts on the integration topic.

And of course, share your thoughts and comments on my article, in the comments section.

Acknowledgment

*) Intro photo by Denny Müller on Unsplash

References

[1] SAP ISA-M: Integration Methodology | Services and Support

[2] Gartner: How to Design Great APIs

[3[ LeanIX: Integration Architecture

[4] LeanIX: Best Practices to Define Data Objects

[5] SAP Cloud Integration: Guidelines to Implement Specific Integration Patterns | SAP Help Portal

[6] EIP: Table of Contents - Enterprise Integration Patterns

[7] MuleSoft: Top five data integration patterns