FormatConversionBean - One Bean to rule them all!

[engswee]

Update 9 Sep 2018: FormatConversionBean now available in CPI as well

Update 19 May 2015: Added new converter Base64EncodeConverter

Update 30 Apr 2015: Added new converter Base64DecodeConverter

Introduction

Over the past months, I have introduced various custom adapter modules that are configurable and reusable; aiming to handle various format conversions to/from XML that are not available in standard adapter modules.

I have recently refactored the different modules and consolidated them into a single module, FormatConversionBean. The benefits of this refactoring are:-

• Single point of entry for the different types of conversion


• Refactoring of factory class utilizing ConverterFactory allows dynamic instantiation of converters


• Easily extensible to new converter formats by extending class AbstractModuleConverter


• Conforms to the Open Close Principle whereby new converters can be added without modification to existing code

Converter Class List

Below is the comprehensive list of the converter classes that are part of FormatConversionBean now. This list will be updated as and when new converter classes are introduced in the future.

Converter Class	Source Format	Target Format	Reference
Excel2XMLTransformer	Excel	XML	ExcelTransformBean Part 1: Convert various Excel formats to simple XML easily
XML2ExcelTransformer	XML	Excel	ExcelTransformBean Part 2: Convert simple XML to various Excel formats easily
DeepPlain2XMLConverter	Deep Plain Text	XML	DeepFCCBean - The better FCC at meeting your deep (structure) needs! (Part 2 - Flat File to Deep XML...
XML2DeepPlainConverter	XML	Deep Plain Text	DeepFCCBean - The better FCC at meeting your deep (structure) needs! (Part 1 - Deep XML to Flat File...
JSON2XMLConverter	JSON	XML	JSONTransformBean Part 1: Converting JSON content to XML
XML2JSONConverter	XML	JSON	JSONTransformBean Part 2: Converting XML to JSON content
Base64DecodeConverter	Base64 (Plain/XML)	Binary	Base64DecodeConverter - Base64 decoding made easy!
Base64EncodeConverter	Binary	Base64 (Plain/XML)	Base64EncodeConverter - Base64 encoding made easy!

Usage of Module in Communication Channel

Module Processing Sequence

Number	Module Name	Type	Module Key
<Depending on position of module in chain> 1) Asynchronous scenario, Normally before the last module in channels 2) Synchronous scenario, Before last module to convert request payload After last module to convert response payload	Custom_AF_Modules/FormatConversionBean	Local Enterprise Bean	<Any Arbitrary Value>

Module Configuration

For configuration of parameters, refer to parameter list of each converter.

Source Code and Deployment Archive

The Java source codes for the module are located in the following GitHub repositories.

GitHub repository for equalize-xpi-modules

The EAR file for deployment (compiled on NWDS 7.31 SP13 Patch 0) is also available from the latest repository release below. The EAR file can be deployed directly on PI/PO systems with versions that are same/higher than 7.31 SP13 or 7.4 SP8.

Latest release for repository equalize-xpi-modules

To set up the EJB/EAR projects on NWDS for further customization, refer to the following.

NWDS version	Reference
7.3x or higher	Setup com.equalize.xpi.af.modules Adapter Module project in NWDS easily using EGit
7.1x	Recompile com.equalize.xpi.af.modules as EJB 2.1 modules in NWDS 7.1x

Further Reference

If you plan to download the source codes into your own NWDS installation to make changes and/or develop new modules, the following blog would be useful for performing testing in NWDS prior to deployment into the PI system.

Standalone testing of Adapter Module in NWDS

Guide on EAR deployment via NWDS

Deploying .EAR files Using NWDS

[avinash_ayanala]

Hi Eng,

It was really great info, the EAR file has been successfully deployed and it is working fine.Thanks a lot.

Regards,

Avinash

[engswee]

Hi Avinash

Thanks for your comment. You are the first to inform me that the modules' EAR file has been successfully deployed and I really appreciate this feedback. It is good to know that it is working on other systems apart from mine.

Rgds

Eng Swee

[Ryan-Crosby]

Nice work Eng Swee!

I don't have any specific needs for this now but I will have to remember it for the future.

Regards,

Ryan Crosby

[Former Member]

Hi engswee.yeoh and avinash.ayanala,

I am working currently on a requirement where the input is proxy and output is a file in excel format. I am using SAP PI 7.31 .

I have some queries related to this.

Is the below the correct EAR that we need to deploy or it contains only the JSON2XML converter. My exact need is to convert XML to EXCEL but I want to try the FormatConversionBean as a whole since it contains the entire module and I would not have to deploy any other separately. So do we need to deploy only the below EAR file?

com.equalize.xpi.af.modules.app.ear 

I will provide feedback and updates once I am done.

Please do let me know.

Thanks,

Shaibayan

[engswee]

Shaibayan

Your queries will be answered directly in the thread below that you opened with the same details.

Output File in Excel format in SFTP server?

Rgds

Eng Swee

[Former Member]

Sure Eng.

Thanks a lot.

[juan_vasquez2]

Hi Eng, can you help me please with this issue

I am using the Adapter Module success, in the receiver channel

but i need to send some header values:

Cache-Control: no-cache

Content-Type: application/json

Application-Secret: 6474b25daaaaaaaaaacc7c9c892195352354facd

where can i send it?

what is the better channel type soap http (axis)

the content is in the right way, you can see in the immage

thanks a lot

[engswee]

Hi Juan

Thanks for your interest in FormatConversionBean. Glad to know that the JSON transformation is working well for you.

As to your query regarding custom HTTP headers, since it's unrelated to this module, I'd suggest that you open a new thread in the forum for this issue instead.

As always, please provide as much details/screenshots as possible in the thread, i.e. PI/PO version & SP, scenario overview & details, adapter types, error messages, etc.

Rgds

Eng Swee

[yeeloon-khoo]

Hi Eng Swee,

Feedback on my trial.

Deployed EAR in PO server 7.31, so far tested excel2xml working well, able to convert accordingly to xml. Nice! Many end users still love excel for manual upload, this will help a lot.

Your blog, example and github code is very useful and inspiring. Thanks again.

Regards,

Yee Loon

[engswee]

Dear Yee Loon

Thanks for your feedback. I am very pleased to hear that is is working well. Please feel free to let me know if you do encounter any issues if you do get around to testing the modules more.

Best regards,

Eng Swee

[former_member186851]

Excellent and very useful blog.

Both Json to XML and XML to Json conversions are working fine.

[engswee]

Hi Raghu

Thanks for the feedback. Glad to know that the conversions are working fine :smile:

Rgds

Eng Swee

[former_member192892]

HI engswee.yeoh,

Firstly, let me congratulate you on the blog. It is awesome and very helpful.

I deployed the module and tested the Deep flat to XML conversion. I wanted to report a small bug. On the variable for fixedFieldLengths since you are doing a number format check after removing commas, the module will throw error any time you exceed 2,147,483,647 which is the maximum value int type supports. May be you can use a string.matches("\\d+?");

Once again, amazing work :smile:

Regards

Varun

[former_member205592]

Hi Eng Swee Yeoh,

Tried DeepPlain2XML and XML2DeepPlain in the File and JMS adapters and they worked great.

Thanks for the detailed blog it was very helpful.

[engswee]

Hi Varun

Thanks for your kind comments and also the feedback regarding the bug.

Would you be able to provide some more details so that I can try to reproduce it? Can you provide screenshots of the module configuration as well as the error you are getting?

Rgds

Eng Swee

[engswee]

Hi Anil

Really pleased to hear that they are working well. :smile:

Rgds

Eng Swee

[former_member192892]

Hi engswee.yeoh,

To reproduce the error, you just need to do an FCC for a fixed field length file and you need to ensure that the recordset.fixedFieldLengths parameter has a value which exceeds 2147483647 after you remove the commas. I think if you test with a file having 5 columns with each column having a fixed length of 25, you should get the error.

The exact location of error is at the class com.equalize.xpi.af.modules.util.RecordTypeParameters on line 54. I removed the try/catch block and used the below snippet. It worked like a charm :smile:

if(!lengthsWithoutComma.matches("\\d+?"))

{

throw new ModuleException("Maintain only integers separated by commas for '" + fieldFixedLengthsName);

}

[engswee]

Hi Varun

Ah, thanks for elaborating further. I've fixed the bug accordingly and published a new release for the EAR file as well as the source code.

Thanks again for reporting the bug. It is much appreciated! :smile:

Rgds

Eng Swee

[vadimklimov]

Hi Eng Swee,

Thank you for developing and sharing this module - very helpful, and your efforts are very much appreciated. Recently we used it for XML to JSON conversion in JMS based integration scenario - worked perfectly well.

One question to you. Manifest contained in a project, specifies implementation vendor ID = "sap.com". As a result, corresponding component gets deployed with vendor = "sap.com", which is normally reserved / used by SAP standard components. Would you suggest downloading source code provided by you, adopting manifest and building deployable, or may you adjust vendor identifier to something different from "sap.com" and rebuild EAR so that it already contains custom vendor identifier?

Regards,

Vadim

[engswee]

Hi Vadim

Thank you for your feedback. I'm really glad to hear that the conversion worked well.

Regarding the Manifest file, to be honest, this is something that I'm not that familiar with. I checked both the EAR and EJB projects - there is no manifest file maintained in the EAR project and the one in the EJB project does not specify the vendor ID (very little info in that file).

It looks like the MANIFEST.MF file within the EAR deployable file is automatically generated when I perform Export > SAP EAR file for the EAR project. I guess I can of course manually edit the entry in the generated EAR file, but do you know if there is some other automatic way to achieve this - maybe manually maintaining a manifest file in the EAR project?

Have you adjusted the vendor identifier before for some custom adapter module that you have developed? Let me know if you do, I'm all up for modifying it accordingly for the EAR files I provide.

Rgds

Eng Swee

[Ryan-Crosby]

Hi Eng Swee,

I was able to achieve this in the past by adding the following tag into the application-j2ee-engine.xml for the EAR file:

<provider-name>idexx.com</provider-name>

Regards,

Ryan Crosby

[vadimklimov]

Hi Eng Swee,

I assume, if manifest files (MANIFEST.MF and SAP_MANIFEST.MF) are not present in project structure, then NWDS auto-generates them when building EJB and EAR projects, and uses standard attribute values (this is where vendor = "sap.com" comes from). Since in built EAR file, after I decompile it, I can see manifest files:

I resolved this issue by explicit creation of manifest files within a project. I didn't create them in NWDS (but it is worth trying this - probably, it will be much simpler), but used build tools and specified what kind of manifest file I would like to assemble when building EJB project. To be more precise, I used Gradle (http://gradle.org/) in order to automate bundling the deployable. In Gradle script, I made use of Java plugin and its property manifest, where I provided attributes and their custom values which I want to put into manifest file. Usage of Gradle purely for this task may be a bit overkilling - but I used Gradle also for dependency management (especially when adapter module / custom job was using external libraries and I wanted to insert some of them, but not all, into the same deployable) and entire compilation and assembly of JARs and EARs, so manifest creation was just one of tasks of executed Gradle script.

Regards,

Vadim

[engswee]

Hi Vadim, Ryan

Thank you so much for both your feedback. I've definitely learnt something new :smile:

I've managed to adjust the value generated in the MANIFEST.MF file by adding the provider name into the application-j2ee-engine.xml file as suggested by Ryan. That worked really well and once it's deployed into the SAP system, the vendor value is reflected correctly in the different views in NWA.

I've updated the source code and also released a new EAR file in the GitHub repository for this change.

Thanks also for the info on Gradle. Will definitely have to look into it at some point! :wink:

Rgds

Eng Swee

[former_member200790]

Thank you

The information provided by you became great help to me.

But i was occur problem in my source of NWDS

APIAccessFactory.java and MessageLogger.java Is not find in MessageLoggerHelper.java

Where is find JAVA Source? and .jar download url?

Regards,

Bang

[engswee]

Hi Bang

Glad to know this was helpful for you.

The pre-packaged Adapter library that comes with NWDS 7.31 are reduced versions and do not contain all classes for some packages. As such, you will need to get the full JAR file for com.sap.aii.af.ms.ifc_api.jar from your PI server.

Refer screenshot below that shows the package and classes that are in that JAR file. This will enable MessageLoggerHelper.java to compile successfully.

Refer to the following Wiki on where to get that file in your PI system.

Where to get the libraries for XI development - Process Integration - SCN Wiki

After you have retrieved that file, create a User Library in NWDS (give it any name) that includes that JAR file. Then add the newly created User Library to the EJB project's build path.

Rgds

Eng Swee

[former_member200790]

Dear Eng

Thank you, kind explanation for me

Is not found MessageLogger.class in com.sap.aii.af.ms.ifc.api.jar in PI 7.3

So, i'm going to after annotation in MessageLogger.class

Regards,

Bang

[Former Member]

Hi Bang,

can i deploy the FormatConversionBean on our PI 7.3 SP10 ?

[Former Member]

Hi Eng,

 

Very nice work with providing us this wholesome solution.

I am planning to use this solution for my below requirement. I have JSON file which i will need to read and update that data into Database tables.

 

So can you please let me know once i read this file using File adapter and custom JSONtoXML converter module, how can i map it to the database tables?

 

Also i will let you know once i deploy your module in our server, we are on SAP PO 7.4 SP9 how does it work out.

 

Thanks,

Hetal

[Former Member]

Any idea how can i deploy the EAR file on SAP PO 7.4? I do not need to make any changes in the code you provided or any enhancements, hence if you can let me know how to deploy it directly on SAP PO.

 

Do i still need to have NWDS to deploy the file on the server?

 

Thanks,

Hetal

[vk_k3]

Hi Eng Swee,

I have been fan of this bean and I have been using this quiet a number of times.

Recently, I ran into trouble when I am using xml2json converter in this bean, when I am passing 800 + rows of XML, total of 1700kb , it takes around 9 minutes to perform the conversion.

Do you think it is normal ?

Regards,

Vk

[former_member245008]

Hi Eng Swee,

I am facing issues in downloading the EAR file from the link provided. For some reason the download just stops. Not sure if there is an issue with github. Is there another location where this can be downloaded?

https://github.com/engswee/equalize-xpi-modules/releases/latest

Best Regards,

Paolo

[former_member245008]

Nevermind. Link seems to be working now. Thank you!

Paolo

[peter_wallner2]

Thank you Eng Swee - it works perfectly. We are on SAP PO 7.5

Best regards, Peter

 

[engswee]

Thanks for the feedback, Peter. Real glad to know that is it working fine, even on the newer version 😉

[former_member235786]

Hi Eng Swee,

 

We got the below error while installing  the com.equalize.xpi.af.modules.app.ear file directly   using SUM.

Are we missing anything?

Error:

Component equalize.com/com.equalize.xpi.af.modules.app cannot be installed. Component com.equalize.xpi.af.modules.app is not part of any existing software components.

 

Thanks in Advance.

 

Regards,

Rajesh.A

[former_member235786]

Hi Peter,

How did you deploy the .ear file?

Did you deploy the .ear file directly using SUM?

 

 

Regards,

Rajesh

[engswee]

This EAR package is not packed as a DC that is part of a Software Component (typical of NWDI-based development).

 

I have never used SUM to deploy it so I suggest you use other methods - Telnet, NWDS, etc.

[peter_wallner2]

Hello Rajesh,

No for deploying - adapter modules or EAR files - I always use the NWDS.

There is a "deploy view" in NWDS you can use also for external files such as EAR files. It works well and is really practical.

BEst regards, Peter

[former_member235786]

Thanks Eng..I was able to deploy it using telnet.

I am getting null error in the channel.



 

Channel Configuration:

 



File loaded :

 



 

 

Regards,

Rajesh.A

 

 

 

 

[engswee]

I'd suggest you try it on a receiver channel first and see if there are any errors logged on it which will provide more details.

[former_member235786]

Hi Eng Swee,

Excel2XMLTransformer is working with xls format but I am getting the null error with 2010 xlsx format.

Regards,

Rajesh.A

[vishu_61140]

Hello Eng Swee,

 

Thank you for this wonderful solution. As in the steps I downloaded the EAR file from GitHub (com.equalize.xpi.af.modules.app.ear) and deployed it directly in my PO system using Deployment perspective of NWDS . I have PO 7.5 system.

I intend to use DynamicAttributeChangeBean. But when I give this in the receiver channel module, the log shows error -

"Unable to obtain EJB home interface [Root exception is java.lang.NoSuchMethodException: com.sap.engine.services.jndi.implclient.ClientContext.create()]"

The Deployment seems to be correct -

 



 

Is there any post step that needs to be done or if I have missed anything.

Thanks in Advance.

Regards

Vikash

 

[engswee]

Try and check in the JNDI Browser and/or Application Modules in NWA to see if the EJB Modules are listed there. I don't have access to a PI system at the moment so can't provide more details.

 

Also, double check your module configuration (sequence, name, type, etc).

[arrezende]

Tank's  engswee.yeoh , this module adapter solved my problem in my current project!

[engswee]

Fantastic! Great to hear that 😉

[dhanvanttri]

I have a Bug to report, when i am converting XML to JSON using the module, £ symbol is getting converted to �. Please help. Thanks in advance.

[former_member223432]

HI Eng,

I am working on Proxy to REST sync interface. I have few input fields which has value '/' in it ex: ( AOUS/12 and some date fields(23/04/2020).

at the JSON output payload it is coming as AOUS\/12 and dates are coming as 23\/04\/2020.

there is an addition of '\' in between. I tried to declare the field as "string" in custom xml/JSON coversion rules. but still its not working. i also tried to declare the field as "Date" in the datatype but still no luck because in the mapping its correctly populated.

the issue is coming at the REST channel while converting XML to JSON format.

Can you please help me how to get rid of this extra '\' at the channel level.

 

thanks

Smith

thanks

Smith

[f_cemunal]

Hi Eng,

First of all thank you for this useful blog, it is very helpful.

I have been dealing with a project which i need to send a zipped file to rest api. My problem is the content of the file must be in json.

I am using rest receiver and "AF_Modules/MessageTransformBean" and "AF_Modules/PayloadZipBean" modules, you can see in screenshots.

can you help me with some advice on this please?

 

[stevelewis]

Hello Eng

Thank you for creating this amazing module. I have been using it successfully to convert relatively small xml files to excel.

In one scenario I am also using the payloadZipBean module to unzip the file before it passes to the FormatConversionBean. All of this works great for small files. However, I have a case with a large xml file(200mb) which causes the Receiver File adapter to hang with the following error:

• Adding message to blacklist. Reason: Message (in delivering state) found during startup


• Message added to blacklist


• Moved message to the non-delivered state. Reason: Blacklisted message (in delivering state) found during startup

It appears the conversion is taking too long for the large file to convert and the message gets stuck in Delivering then blacklisted. If I remove the FormatConversionBean I am able to process the file successfully as an xml. Do we have a known fix for this use case?

Thanks for reading and I look forward to your reply.

Thanks

Steve

[engswee]

Hi Steve,

 

200MB is definitely a big file, and the module has never been tested for such use case. Maybe the memory handling is not optimised for it.

 

I am no longer actively working with PI, so I would suggest that you clone the Git repository and try to test it locally, and make any fixes as required.

 

All the best!

Eng Swee