Using SAP PI/PO Directory API In Complex Carveout Scenario

[MatthiasFricke]

Intro

This article is mainly a discussion of challenges and problems I was facing on a recent "carveout" project. 

To accomplish my tasks, it turned out, that I had to do a lot of Java coding, and to a large extend, this article focuses on the presentation of that work.  

Though it is not mandatory, a sound knowledge of programming, preferably in the Java language, is helpful, especially as I do not have time / resources to document all the features of the work. (And wrong usage can lead to undesired results.).

 I have structured the article as follows:

1. Disclaimer
2. Overview: Discussion of the overall problems, introduction into main task.
3. SAP PO Directory API: Short Introduction.
4. Code and Use Case Intro: Intro on the code, explain one of the main use cases. 
5. SAP PO Migration Cockpit: This is just a short excursion to explain why it had not been used.
6. Use Case and Program: Introduction into the program in the light of the use case.
7. Development setup.
8. Epilogue

Disclaimer

No responsibility for usage of the work discussed here will be taken. 

Especially ill usage of the program/code discussed here can lead to undesired results.

Be careful, and study the sources which are under Apache 2.0 license before using the program. 

Overview

The project setup was as follows:

A complete SAP system landscape had to be copied and configured according to the needs of the new entity.

This involved a copy of SAP PI/PO (AEX/AAE) systems:

SAP ERP Source	SAP ERP Target	SAP PI/PO Source	SAP PI/PO Target
ER1	ER4	AE1	AE4
ER2	ER5	AE2	AE5
ER3	-	AE3	-
		AX0 (Central Instance)	AX1 (Central Instance)

My task was to adapt the PI/PI directory artefacts (for the copied systems) to the new landscape.

SAP PO Directory API

To accomplish my tasks I made use of the SAP PO Directory API:

The API consists of a set of SOAP web services, which can be used for create/read/update/delete (CRUD) operations on various directory objects. If you do not know the API yet, I recommend using  SoapUI to gain familiarity with the XML representations of the usage of these services (See reference blog below).

I will not enter into the basic usage concepts of the directory API, as there are already various blogs available for this. This is just a small selection:

Purpose	Link
Official documentation	SAP PO directory API
Introduction of Directory API	OVERVIEW-ON-PI-PO-APIS
Usage of Directory API (SoapUI)	SoapUI-Usage

Code and Use Case Intro

As often, the code has evolved out of specific needs, which changed during various stages. It's maturity level is a snapshot from the end of the project, so do not expect to find a ready to use program, but have a look at your needs, compare them with the code you find here, change it where necessary, try it.

Especially the many command line parameters might cause problems and in worst case damage, if not used carefully. In this blog I will only describe one use case I was facing.  I do not have the resources to get into other use cases here.

The code is under Apache 2.0 license, bevore use, please make yourself familiar with its meaning.   

But before we dive into the program, let's first look into the problem.

The following table gives an idea on the overall number of directory objects in the source PI/PO landscape:

Object	Number
Business Component/System	1683
Integrated Configuration	4458
Communication Channel	10260

The existing PI landscape looked as follows:

The carveout system PI landscape looked like this:

(Note that the arrows should be bidirectional)

Distribution of communication channels per adapter engine (Source system landscape):

Instance	Number of channels
Central Instance AEX	1441
AE1	5069
AE2	2163
AE3	1587

The chosen migration strategy from existing landscape to carveout landscape was to do a system copy for the central instance AX0 as well as AE1 and AE2:

System Origin	System Target
AX0	AX1
AE1	AE4
AE2	AE5

AE3 of source landscape was not needed in target carveout landscape.

The first task after the system copy was to adjust the adapter engine names of the communication channel objects originating from decentral adapter engines AE1 and AE2, so that the hostname of channel objects from AE1 point to AE4, and the ones from AE2 point to AE5. 

Here is an XML snippet showing the property location of the AE name for a communication channel after the system copy (Read operation on CC):

I found it necessary to change the AE name for each channel sitting in a decentral engines AE1/AE2, so that that it would point to the new decentral engines AE4/AE5.

The background for this is, that the object data of the integration directory is stored on the central instance AE. After each update, the data gets propagated to the decentral AE via CPA cache notification. As long as the CC AE names are all obsolete (pointing to the old decentral AE's), this mechanism can not work. That's why it is necessary to assign the correct new AE name to the corresponding CC's.

Here is an example on how it has to be done (Change operation on CC). The old AE name af.ae1.aehost01 gets replaced by af.ae4.aehost04:

SAP PO Migration Cockpit

Before I started writing code myself, I had a look at the SAP PI/PO migration cockpit, which comes with each PI/PO installation. (http://<hostname>:<port>/webdynpro/resources/sap.com/tc~pi~tools~dirmig~wd/DirectoryCockpit#)

The tool is generally capable of carrying out the change. But the selection is not sufficient. Here is a screenshot of the selection options of the migration tool:

Selection of channels per AE is not possible. So, this tool could only be used if all channels (or at least components) would contain some sort of AE reference in their names.

As for my situation this was not the case, I resolved to write code to carry out the change.

Use Case and Program

But there is an additional problem than just the selection per adapter engine (AE):

The problem is not the change of the AE name itself, but the activation of the resulting changelist. The changelist is checked for consistency during activation, and if a given CC has been updated to a new AE name and this CC is sitting inside an ICO, then this ICO does get checked for consistency. An activation error will be generated, if other CC('s) sitting inside this ICO do still point to the old AE. So, you have to change all CC AE names for all channels inside an ICO in one changelist. 

To overcome this problem, I started with the simple approach of writing code for an update of all channels for one AE. The Idea to update all channels of a given AE in one big change list is theoretically stringent, but it turned out to be practically not feasible due to various ICO inconsistencies, mainly for missing mappings in multi receiver, and likewise multi interface, ICO's. (The system landscape had grown over the years and  contained a lot of ICO's with many receiver interfaces with some mappings missing.)

This meant that I had to follow a different approach.

The Idea was to select ICO's which belonged to a given adapter engine (AE), or with other words, ICO's referencing communication channels of AE1 / AE2, so that the program would always run only for one adapter engine. 

Once a set of ICO's had been selected for one AE (via old AE id), the program would proceed as follows.

Loop over the ICO selection and for each ICO:

1. Select all channels it is referencing (could be in the dozens).
2. Change the AE name for each channel.
3. Check the resulting change list for consitency:
   • If changelist is consistent, activate it imediately.
   • If changelist is not consistent due to some channels beeing used in other ICO(s) which still point to old AE, recurse the step for each referenced ICO. 
   • If changelist is not consistent due to other errors, release the changelist proceed to the next ICO.

If a CC is used in more than one ICO, than that ICO has to visited also and so on (the program has to recurse through the dependency tree).

The program can be executed as an executable Java archive from the command line, it has no gui interface. 

A run configuration for the above described problem would look somewhat like this:

java -jar PO-API-0.0.1-SNAPSHOT.jar -process ccico -sourceUser "Basic dXNlcjpwYXNzd29yZA" -targetUser "Basic dXNlcjpwYXNzd29yZA" -sourceUrl http://<pihost>:<port>/ -targetUrl http://<pihost>:<port>/ -ccSetInactive false -isReadonly false -isAllLogs true -icoIsChangeAAE true -ccHostChange false -icoIsActivate true -icoCheckAaeName ae1 -icoNewAeHostName aehost4 -icoNewAeID ae4 -icoCount 3000 -icoIsTouch false

Let's look into the meaning of the command line arguments for this use case:

Parameter	Explanation
-process ccico	Process branch ccico means: Select ICO's  For each ICO process cc's
-sourceUser "Basic dXNlcjpwYXNzd29yZA"	User:Password (Base64)
-targetUser "Basic dXNlcjpwYXNzd29yZA"	User:Password (Base64)
-sourceUrl http://<host>:port/	Url of source PI/PO
-targetUrl http://<host>:port/	Url of target PI/PO
-ccSetInactive false	Deactivation of processed CC's
-isReadonly false	Do not carry out changes
-isAllLogs true	Log everything
-isChangeAAE true	replace the current AE name with name of new AE
-icoIsActivate true	Activate change lists as soon as possible during program run. If set to false, many open changelists might result, which might be undesirable.
-icoCheckAaeName ae1	System id of old AE
-icoNewAeHostName aehost4	Hostname of new AE
-icoNewAeID ae4	System id of new AE
-icoCount 3000	Limit the number of ICO's
-icoIsTouch false	Update processed ICO's (obsolete, not necessary)

You might wonder about the parameters sourceUser, targetUser, sourceUrl, targetUrl. As the program can be used to copy directory objects from one PI/PO system to another one, these parameters are necessary. But for the use case explained here, this is not the issue, so the values will be the same for source and target parameters.

This run configuration would select up to 3000 ICO's related to adapter engine ae1 and change the AE name for each CC referenced by each ICO.

Note that in this use case (for other use cases it is not recommended to activate any changes during runtime) it is important to activate each changelist immediately during runtime as soon as being consistent. 

If an ICO contains CC's  which in turn are used in other ICO's, then these ICO's have to be processed as well and added to the changelist. This is done recursively. The bottom out condition is a consistent changelist.

This behaviour might slow down the program, because the consistency check for the changelist will take longer and longer with a growing number of CC inside it. I had situations where the recursion level was deep and the program seemed to hang.

Again, just for this use case: Once the changelist is consistent, it gets activated immediately at runtime. The program then continues to the next ICO as long there are any left in the loop.

The parameter "icoCount" is there to limit the initial selection of ICO's.

For a start/test this can be set 1. The program would then just select one ICO, and process it's CC's. (And all the CC's of other ICO's if there are dependencies between them.)

There are the following additional parameters which can be used to limit the initial selection set of objects to be processed:

Parameter	Explanation
-icoSenderPartyID partyXY	Limit ICO selection to (sender) partyXY
-icoSenderComponentID componentXY	Limit ICO selection to (sender) componentXY
-icoInterfaceName interfaceXY	Limit ICO selection to (sender) interfaceXY
-icoInterfaceNamespace urn:xyz	Limit ICO selection to (sender namespace) urn:xyz
-icoRecParty receiverPartyXY	Limit ICO selection to (receiver) receiverPartyXY

It is always a good idea to start testing the program with a small selection of Directory objects. This can be done for the other process options as well, have a look to the source code to check these selection limiting parameters.

The following is an overview on other main processing branches of the program, which are not discussed in depth here. (Visit the source code/class diagram to get an inside)

Main branches of the program configured with command line parameter "process":

Processing option (*)	Description
cc	Process CC's (Direct selection of CC)
ccico	Process CC's (Selection via ICO's)
cccount	Process CC's (Count CC's per AE)
ccdel	Deletion of CC's (Selection via ICO's)
ccdeldirect	Deletion of CC's (Direct selection of CC's)
ccdelicolist	Deletion of CC's (Usage of whitelist file)

These processing options have to be used in combination with other command line parameters. (Again, be careful)

Here is an overview:

Parameter (*)	Description
sourceUrl	Source system URL
targetUrl	Target system URL
sourceSystem	Replace action source
targetSystem	Replace action target
businessSystem	Obsolete
isReplaceSystemName	Replace source with target (see above)
sourceUser	base64 encoded user:password of source system
targetUser	base64 encoded user:password of target system
sourceClient	Replace SAP client in name source
targetClient	Replace SAP client in name source
icoSenderPartyID	Limit selection to one party specified here e.g EDI
icoSenderComponentID	Limit selection to one component specified here e.g BC_ERP_CLNT_001
icoInterfaceName	Limit selection to one interface specified here e.g Dummy
icoInterfaceNamespace	Limit selection to one namespace specified here e.g urn:test:eu
icoRecParty	Limit selection to one party specified here e.g EDI
icoReplaceSenderSystem	Replace the sender e.g BC_NEW_SENDER
icoReplaceReceiverSystem	Replace the receiver e.g BC_NEW_RECV
icoReplaceSenderInterface	Replace the sender interface e.g Dummy_new
icoReplaceSenderInterfaceNamespace	Replace the sender interface namespace e.g urn:test:eu
icoIsDownload	Download ICO's as XML
icoIsDelete	Delete ICO's
icoIsCreateFromFile	Create ICO's from XML
icoTouch	Update ICO's belonging to changed cc (obsolete)
icoIsActivate	Activate changelist's during processing
icoCheckAaeName	Select only ICO's for a specific AE id eg. ae1
icoIsChangeAAE	Change AE Name from ICO e.g. true/false
icoNewAeID	New AE ID eg. ae4
icoNewAeHostName	New AE name eg. aehost4
icoCount	Count ICO's
icoIsTouch	Update ICO's belonging to changed cc (obsolete)
icoWhiteListFile	Name of Whitelist XML file
icoIsMoveAeToCentral	Move channels od ICO's to cental AE
icoIsCheckAllCcInactive	Check if all channels for a given ICO are in state inactive
icoReplaceAction	ICO related replacement of sender component
ccCheckAaeName	AE name e.g. ae1
ccSetInactive	Set cc to inactive e.g.true/false
ccPartyID	Limit selection of cc to one party specified here e.g EDI
ccComponentID	Limit selection of cc to one component specified here e.g BC_ERP_CLNT_001
ccHostChange	In CC config add string UNKNOWN to hostname/ip e.g. true/false
ccID	Limit selection of cc to one cc specified here e.g SOAP_Sender
ccdelUseWhitelist	Deletion of cc's use whitelist for cc's not to be deleted e.g. true/false
ccIsCopy	Copy cc
bcPartyID	Limit selection to one party specified here e.g EDI
bcComponentID	Limit selection to one component specified here e.g BC_Sender
isReadonly	Do not carry out changes
changeListID	Changes go in this changelist
isBackwards	process list in reverse order
isAllLogs	Log maximum true/false

* All these command line parameters have to be prepended with a minus.

Special care should be taken for the parameter changeListID. 

If used, changes go into the changelist specified.

For usage, a changelist has to be created manually in ID beforehand.

Usage of changelist ID like this:

-changeListID 19946677f8ea11eeb8b10000001f74ee

If not used, then each change will go into a new changelist, which can be desired (see discussed use case above).

On the other hand, there are use cases where usage of a common changelist is highly recommended.  For example there the use case of a inactivation of CC's can be carried out. If no changelist ID gets specified for this use case, then each change will end up in an own system generated changelist. This could lead to many open changelists (and a lot of manual work), depending on the number of open changelists. 

Development Setup

For the development setup you need jdk 1.8, Apache Maven build environment as well as git integration.

If not yet installed, there are various options to get jdk 1.8, here is just a selection:

Vendor/Project	Url
Zulu (Free)	https://www.azul.com/downloads/#downloads-table-zulu
Oracle (You have to register)	https://www.oracle.com/java/technologies/downloads/
SAP (S-User necessary)	https://me.sap.com/softwarecenter  (search for SAPJVM8)

The corresponding maven java project can be checked out from github:

git clone https://github.com/sapstern/PO-Directory-API

To build proceed on the command line as follows:

cd PO-Directory-API/PO-API

mvn clean install

if mvn finished successful, you find the executable jar under:

/target/PO-API-0.0.1-SNAPSHOT.jar.

You find a class diagram in the root folder, which you can consult in order to get an overview.

The startup class holding the main method is com.sapstern.po.api.v2.ApiProcessor

All command line args are defined in class com.sapstern.po.api.v2.abs.AbstractAPiHandler.

If you do spot any errors, have questions, etc, do not hesitate to contact me.

Epilogue

From my side, for now, thats it. As already mentioned, there are many more functions the program can perform. I simply do not have the time to document them here. Look at the code, try them out, and of course you can ask me.