Why Integration is Easier and Harder than Ever Before

[former_member182046]

A few days ago, I had a conversation about the challenges ABAPers are facing and about current trends with respect to these challenges and the skills developers need to meet them. The person I was talking to mentioned that integration was specifically brought up as a key issue: Apparently, it is something that developers struggle with and that poses them great difficulties. My conversation partner asked why integration was more difficult today than it used to be.

My response was, it is so hard because it has never been easier. In a nutshell, when integrating systems, user expectations set the bar higher than ever before.

“My god, it's full of silos!”

In the past, users understood that each IT system was a silo:

• Their login would be “I30540003” in on system, “MILLERTHEO” in another, “TMILLER0001” in the third system, etc.
• Each system had its own client software that you would have to launch from your desktop. SAPGUI here, a Borland Delphi windows client executable there, a Telnet TN3270 emulation connecting to an IBM host system there. Maybe not even functioning copy-paste between windows.
• Each system would have its own database, with long time spans between synchronization: A nightly file download and upload here, weekly synch there. A change in one system would show up in another systems days later.

Current expectations

Today, when using two different pieces of software that are at least remotely related, what we expect is roughly this:

• Real-time backend data integration: All data must be perfectly consistent with other data we consume from other related sources, in real-time.
• Comfortable single sign-on: It’s not user-friendly to require a user to logon as long as there is a valid logon with some other system which could vouch for them.
• Use identities from other system: Why should I create usernames and passwords with new systems if they could just accept my Facebook/Google/Windows domain identity and credentials?
• Cross-system personalization: Why should I have to teach two or three systems the same about what I like and what I don’t? Can’t they inform each other about these things?
• Seamless navigation: If a screen flow takes me from one system to the next, it should be a seamless and consistent navigation flow with, at the very least, properly functioning back and forth navigation, and ideally a consistent look and feel, screen layout patterns, icon language, etc.
• Consistent process flow and monitoring: Flows across multiple systems should be as robust and easy to monitor as flows within a single system. If something goes wrong, I want to know what went wrong and how, and how to get it going again.

From the point of view of an architect or software developer, there are even more requirements:

• Architectures should be designed so that there is the least possible amount of redundant development. Business logic, screen fragments, and data structures that are needed in two different applications (or IT systems) should somehow be built once and reused wherever else required. It doesn’t matter if one environment is a native iPhone application, a second environment is an offline HTML5 page running in a web browser, and a third environment is an ABAP background processing a nightly job – people already become nervous when even a trivial chunk of business logic such as “birthdate must not be initial” is implemented redundantly across these platforms.
• As the functionalities required by a single end-to-end process are spread across multiple systems, we have to find solutions in case one of these systems is temporarily down. So we have to use mechanisms like message queues, make sure that messages are sent exactly ones or that a web service that handles them is idempotent, we have to be able to monitor the message flow associated with a business process and track and correct errors, and maybe our process flows need to have a Plan B in case a required subsystem is down so we can tell the user: “Your entry has been submitted and is pending approval, you will get a notification as soon as it has been processed,” instead of giving them the usual instantaneous feedback.
• Applications should be maintenance-friendly: Even though the steps of an end-to-end process are spread across multiple systems and platforms, changes in one platform should not require adaptations in the other systems (or break them).

Fig.: The bar is high

Who raised the bar?

Putting it all together, we can see that the bar with respect to seamless frontend and backend integration is infinitely higher today than it used to be. But why? The answer lies in another question and answer: Why did the mountaineer climb the mountain? – Because it exists. Why did the architect create a tight coupling between the systems? – Because there’s a way to do it.

From silos to a service web

The first driver is SOA: In enterprise IT, the siloed landscapes were shaken up when SOA came along and it was possible to call functions provided by one system from another. This led to a reduction in redundancy: Functions that used to have many implementations on different platforms in a company’s application landscape were reduced to one implementation, which can be called as a web service by other systems. This possibility has changed people’s thinking and the trend has just been going on and on.

Internet experience as a driver of change

The second driver is the user experience people have with Google and Facebook. Think about how many Google sites you use: Out of Calendar, Gmail, Search, News, Drive, Music, Movies, Maps, there are certainly going to be a few you will use on a regular basis. They all log you on transparently and automatically, without asking for a password or username, and you never have to tell them anything twice. I don’t even remember my Google password, but I use my Google account on a half dozen sites almost daily.

It’s similar with Facebook. I don’t use any sub-sites of Facebook but whenever a new non-Facebook site I want to use gives me the choice between “Sign up” (create a new username and password combination that tortures me from now on) and “Log in with Facebook,” I certainly know what I do. Facebook offers great value to me, and many other users, as a leading identity provider for third-party web sites.

Users take these expectations to work and demand that the enterprise IT systems they work with live up to the same standards of simplicity and user-friendliness. They want to have as few identities as possible, and be able to use them with as many systems as possible.

Availability of standards

Getting back to the mountaineer who climbed the mountain because it existed – the availability of wide-spread IT standards for integration put us in a similar position. Single sign-on between your new custom system and an existing system running anywhere is possible thanks to open standards – and so it is seriously considered and added to your new project’s backlist.

Near real-time data replication between systems is easy to implement with SAP LT Replication, and so it is considered and possibly added to your project’s backlog.

The same goes for other features such as sophisticated message queuing while a system is down, provisioning data and functions to mobile clients, secure login for mobile devices through the internet, navigation between cloud-based and on-premise system, seamless user integration across companies (e.g. employees of other companies whose identities are maintained in that other company’s system signing on to your  partner-facing portal, with single sign-on from their Windows domain).

New state of the art

The bottom line is that the state of the art has changed greatly, and the level of integration that was normal five years doesn’t cut it anymore today. An architect or developer who is tasked with creating that type of integration may find that they have to read up on internet standards, and get their feet wet learning how to actually implement use them in a real system.

This, I believe, explains why developers are struggling with integration now more than ever before. There may have been a time when IT departments were in the driver’s seat – but now we live in a consumerized enterprise IT world. New tools exist, and users demand that we use them. We have to learn how and do it.