NetWeaver Gateway and PI - There's a place for both!

[sufw]

Now  that a few months have passed since SAP released NetWeaver Gateway, and  we’re starting to see more implementations of it in the wild, I’m seeing more and more john.appleby3/blog/2012/01/09/how-to-architect-sap-netweaver-gateway-for-dummies-and-for-experts in the community about the finer points of Gateway and REST in general.

One opinion I have come across is that NetWeaver Gateway should be merged with PI rather than remain as a separate product with its own technology  stack. That way, PI will remain SAP’s “go to” integration hub and customers can avoid standing up further systems and the maintenance overhead that entails.

As a relatively recent convert to the ‘religion’ of RESTafarianism, and having done a bit of PI work over the past few years, I of course have an opinion here: “Don’t do it!”.

Let me explain:

REST is not a Protocol

REST is an architectural style; a way of architecting system interfaces rather than a formal protocol which one can implement. REST defines a set of principles and constraints. When we design interfaces within these boundaries, then we have something which is RESTful. This can take many shapes of course, as there is no check list or W3C standards document or reference implementation which someone can comply with.

SOA is another, different architectural style. It requires system interfaces to be self-contained services, with defined inputs and outputs, which are separable, minimise side effects, etc. It does not require per se the use of Enterprise Service Buses, UDDI directories, etc. - these things usually appear in specific implementations of an architecture following the SOA style.

There are other architectural styles pertaining to system integration of course. Message Oriented styles come to mind, and there are others which are (unfortunately?) still used such as database integration. In fact, Wikipedia has a list of some.

Of Adapters and Middlewares

SAP NetWeaver PI has been designed from the ground up as a message oriented middleware product with some ESB features. Over time, these ESB  features have grown as SOA architectures became more common and desirable, and that’s A Good Thing. A good middleware product can speak as many protocols as possible, because there are many different implementations of those message-oriented architectural principles. It’s also likely that your middleware product will need to talk to another middleware product, and lots of protocol adapters help bridge the divides between two different implementations of the same architectural style.

But REST is not a protocol, and that is why the idea of a “REST adapter” doesn’t make sense to me. How do you write an adapter into a different  architecture? If all you need is to simply send one request to one RESTful endpoint, PI already has the HTTP adapter which will be perfectly suitable for some simple scenarios. But beyond that, the fundamental differences between the SOA and MOM (Message-Oriented Middleware) architectural styles in  PI’s DNA, and those of a good REST API, will be simply too great to bridge with an adapter.

Let me try to explain this gap by way of some examples:

• REST talks about clients and servers, and doesn’t talk about middleware. PI is middleware.
• RESTful APIs present hyperlinks to clients, which follow the links to modify state. How would PI do this?
• REST mandates no content format. In fact, it allows any content format you can think of. PI really only handles XML.
• REST requires clients to keep state in interactions with the server. PI is  not good at that. (NW BPM will probably change this. NetWeaver BPM as a  REST adapter? Now we might be onto something...)
• PI works best with asynchronous processing. REST does away with abstraction layers providing asynchronicity and relies on natural request/response mechanisms.

Now, I do think that PI should have the ability to interact with RESTful APIs; as middleware, its job is to talk to as many other things as possible. But the fundamental differences between these two worlds will preclude such an approach from being comprehensive or good for all use cases. It may work for really simple things, but it won’t be nearly enough to position PI as the “REST Adapter” into SAP systems. And that’s okay. We don’t need one integration hub to rule them all!

For my money, the NetWeaver Gateway team made the right architectural choices: hosted on an ABAP stack and with good tooling for accessing BAPIs,  custom code, database tables and other APIs internal to an ABAP-based SAP system, rather than loosely coupled to the backend, hosted on a Java stack, or designed around existing functionality of existing products.

That’s not to say that Gateway is perfect, and there are some features which I would dearly love to see in the product. But in my opinion, SAP got the basics right by avoiding the temptation to make PI do something which it wasn’t designed for.

Image by Derick Bailey. Thanks for the Creative Commons license mate!

[MichalKrawczyk]

Hi,

1. nice post 🙂

2. I agree even though I will keep on posting some workarounds for PI to enable parts of REST integration 🙂

Regards,
Michal Krawczyk

[Shabarish_Nair]

REST talks about clients and servers, and doesn’t talk about middleware. PI is middleware.
RESTful APIs present hyperlinks to clients, which follow the links to modify state. How would PI do this?
>>
My assumption was that a REST adapter will have this intelligence.

REST mandates no content format. In fact, it allows any content format you can think of. PI really only handles XML.
>>>
Even though natively PI prefers to have XML, it can still pretty much handle most of the other content

REST requires clients to keep state in interactions with the server. PI is not good at that. (NW BPM will probably change this. NetWeaver BPM as a REST adapter? Now we might be onto something...)
>>>

cant existing ccBPM or NW BPM can help achieve this

PI works best with asynchronous processing. REST does away with abstraction layers providing asynchronicity and relies on natural request/response mechanisms.
>>>>

I am not sure about this statement. Will you care to ellaborate? If you are stating PI finds sync message processing difficult, I will love to see SAP's reaction to this 🙂

-----

I did comment on the earlier discussion around NW Gateway and I still cant help but come back to the point that I am misled at the strategy from SAP around provisioning a comprehensive Integration suite. Are they on the correct track?

By adding additional products and making them independent of PI or rather not plugged into PI, I strongly believe that it will only confuse the customers.

Is it is only a matter of proper allignment and embedding all this into a commmon suite of products or maybe just mere marketing?

Shabz

[sufw]

Hi Michal,

thanks! And please do keep posting. I do agree there is a lot of value in PI being able to integrate with as many protocols and paradigms as possible. But I disagree that it could/should be the best at everything, and it seems we agree?

Sascha

[sufw]

Hi Shabz,

first of all thank you for taking the time to write such a thoughtful response.

Regarding the REST Adapter, I would be hesitant to burden this piece with the "intelligence" to be able to follow links (especially when their representations are highly dependent on the payload format and semantics), or maintain state between several HTTP requests. This is further complicated by the fact that adapters operate essentially outside the standard PI persistence, transaction and monitoring framework. Let's say the adapter needs to send 2 HTTP requests, but the first fails. How would one easily recover into a consistent state? Not nice...


Yes, PI can handle formats other than XML, but it doesn't do so elegantly. For example, Lean Message Search and content-based routing rely on XPath expressions. Using non-XML formats for the payloads make things tricky.


From an architectural perspective, I won't go into ccBPM as it's essentially been deprecated. NW BPM could probably fulfil the requirements of a proper REST adapter as it ticks all the boxes on long-running stateful processes. I have yet to investigate this in detail, but it's certainly an intriguing possibility. However, I wouldn't need PI for the communication (at least not on the RESTful side) as I'm sure it is capable of making HTTP calls by itself.


I know PI can do synchronous interfaces, but I try to avoid having our system do any sync processing unless I have to. In my opinion mixing sync and async interfaces has the ability to negatively affect the whole system's scalability. Sync calls tie up threads in your thread pool for potentially a long time; these are rare commodities as they are tied to the number of CPUs in your system. We're running on IBM p-Series, and CPUs cost more than a small car...

This is especially important for AAE deployments as messages are now only processed by threads from the Sender-side thread pool and the Receiver-side pool is essentially no longer used. Slowing down thread processing (e.g. by making sync calls to a slow backend), will cause a large number of threads to block waiting for I/O, and then potentially impact the system's ability to process messages for other interfaces.

Our PI 7.11 system need to process up to 85 messages/sec asynchronously and we have shown in performance testing that Receiver-side performance is very important and no longer decoupled from sender-side performance in the PI system. If a Receiver is slow to provide a 200 ACK for async messages, it holds threads longer which starves the Sender side of threads in the pool, causing a backlog.

Anyways, what I was getting at is that PI isn't very well suited to handling long-running stateful (sync) processes on a massively parallel scale when you need high performance. That's fine by me, as there are other options. I believe NW BPM could be one, but I haven't looked into that yet.


I do think SAP's product architecture in this regard is sound. SOA and REST is too different to have one unifying product which does both in a meaningful way. By contrast, I have some reservations about their story in the mobile space, but that's a different soap box 🙂

Hope this helped to further explain my thinking on this topic. Thank you for your good questions, and congrats if you made it this far through my waffle 🙂

Sascha

[Former Member]

Sasha,
Thanks for writing this. The first time I read about the gateway tool, the question I had was... Now, why is this a separate tool, instead of being part of PI.

But you had hit the nail on the head, why gateway should not be part of PI. At least not yet. Even with  nw-BPM. PI, in its current state has not yet established itself as the leading middle-ware product. Having the REST capabilities will stress the product.

But eventually, I think PI will definitely scale up and evolve into something else that can absorb gateway, as well as... mobility (but as you said that a different topic altogether)

Regards,
Salai.

[henrique_pinto]

it has nothing to do with product matureness...

[henrique_pinto]

Stateful could be achieved easily with BPM + EOIO processing.

Also, talking about PI & BPM as separate entities doesn't make much sense anymore. BPM, IE, ESR and AEX are rather components of Process Orchestration (PO).

In general, I do agree with your view that full-REST compliant interfacing shouldn't be on PI. As much as full-SOA enablement was not PI. It is within the business solutions, and we know it as Enteprise Services. Of course, PI can converse with these and external SOAP-based interfaces natively, but is not required to do so for a mere SOA-enablement. You don't use PI for that, you use it for A2A complex integration scenarios where you want central monitoring and error recovery capabilities.

In the same sense, you shouldn't need an ESB for REST enablement of the SAP Business Suite. As for the Enterprise Services, this should be part of the core of the business solutions. Rather than being a separate product, it could be a better design to have this REST compliant communication capability as part of the underlying SAP_BASIS component and to have the RESTful services implementations as part of the several application components (in the same sense it was for the ESs).

That being said, for a state-of-the-art ESB, you always want to be able to integrate no matter what to whatever. Achieving REST integration through PI shouldn't be hard if you consider a servlet for URL generation/interpretation, a comprehensive adapter for XML/JSON* conversion (internally, PI will always be based on XML, no matter to which protocol it talks to) and BPM for stateful scenarios.

I don't remember SAP questioning SOAP over PI versus Enterprise Services implementation. Why are we having this discussion now?? SAP should do both REST-enablement of its services (and maybe BOL could be leveraged to make it easier) as well as adding REST conversation capabilities to PO.

One might want to remember that Gateway is not compatible with all things REST. It is compatible with OData, which is by itself a subset (or restriction) of possibilities that REST brings. Out of the box, you can consider that Gateway just supports OData XML, not JSON.

[sufw]

Hi Henrique,

wow, great response!

I very much agree with your notions about how the REST enablement should be an inherent part of the Business Suite in the same way that SOA enablement is with enterprise services. Or at least in the case of ES enablement packages, very closely coupled with the Suite and its functionality.

I also wholeheartedly agree with your comment that Gateway !=== REST, i.e. it offers a subset of what is possible to be built using REST principles. A very good insight which seems easily lost in the current discussions (dare I say hype?) on this topic.

I'm however not fully on board (yet) with the Process Orchestration comments. My entirely subjective and cynical impression is that this is still more at the slideware/marketing level: The three products (PI, BPM, BRM) run on the same Java stack and share some touch points, rather than being a truly integrated suite. However, I have not had hands-on time with the PO solution and will be more than happy to be told that I'm wrong here.

Thank you for contributing to this discussion!

Regards,

Sascha

[henrique_pinto]

But ccBPM was more or less the same, i.e. ABAP-based workflows running on the same instance of PI and with a shared common adapter (ccBPMs talked XI Proxy natively). As far as integration was concerned, it could use the same alerts and mappings as the integration engine, and that was about it.

For BPM within PO, i.e. PI/BPM 7.31, they've added the native PI adapter to BPM. Now, when creating a service interface, you can connect natively to RFCs, SOAP Web Services and PI (i.e. XI Proxy). Also, mappings built on ESR can be consumed natively within "PI" (i.e. integration design) or within "BPM" (i.e. process composer). And to top that up, all monitoring capabilities for whichever components are being centralized on SolMan.

So, yes, there is indeed a single product on the horizon. 🙂

Cheers, mate.

[sufw]

Thank you for the great information! I have only had a little bit of hands-on exposure to BPM 7.2 whose integration with PI was still very much "arms' length". Great to see this is really getting more integrated!

Sascha

[Former Member]

Hi,

I think the positioning of Gateway and PI is in a gray area, especially for those who already have PI.  The blog, http://scn.sap.com/community/pi-and-soa-middleware/blog/2012/03/21/rest-services-with-gateway-and-pi..., discussed some of the issues.

In summary, this is why:

• Most, if not all, of the PI customers have sync services.  Many of them are very high volume.
• A PI REST adapter is now available from Advantco, which fills many of the gaps you described using the HTTP adapter, e.g. provide a link to update, no content, working with non-XML payload, etc.
• Based on feedback, many customers also want to provision non-SAP applications, and question why they need to use another product such as Gateway just for SAP applications, while PI can also meet that need.

For customers who have neither Gateway nor PI, the decision is a lot easier.

Regards,

Bill

[HoHi-cbs]

Hi guys,

very interesting discussion! There are good reasons to have a separate platform or at least runtime for RESTful services, but also some use cases to implement on PI basis.

But my concern about these two completely independent solutions refers to the design time and governance. Instead of having NW Gateway for RESTful services and the ESR used by PI and CE/BPM for enterprise services and other interfaces, there should be the possibility to have one central repository for all services and interface structures independently whether they are RESTful or not.

Regards, Holger

[nabendu_sen]

Nice information sufw. Thanks for sharing.

But I completely agree with shabarish.vijayakumar , if we make available so many things which have common area of applications, it may confuse customers and may be doing already. They need some compact middleware to implement a solution end to end. SAP has so many offerings, sometimes it does not help.

Already enough arguments we have I think. ABAP Stack / Java Stack, ccBPM / NW BPM, REST SAP Adapter / REST Advantco, Seeburger B2B / SAP B2B....list goes on....

From my perspective, WebMethods or Tibco always provided better visibility for future enhancements or features in a consistent manner in Integration Platform.

Regards,

Nabendu.

[Shabarish_Nair]

I do see Gateway converging into SAP PO and this is the logical evolution :smile:

SAP has been creating silo products and now they seem to have woken up to look at a portfolio consolidation.

Reference:

Integration && Orchestration 2014 - A Report Card from Berlin

[nabendu_sen]

Great to hear. Let's see how it goes.

Regards,

Nabendu.