DeepFCCBean - The better FCC at meeting your deep (structure) needs! (Part 2 - Flat File to Deep XML)

[engswee]

Update 2 Oct 2018: New parameter endSeparator to define end of line separator for input file.

Update 4 Sep 2018: Now available in CPI as well.

Update 22 Jan 2016: New parameters missingLastFields and additionalLastFields to handle structure deviations between configured parameters and actual file content.

Update 26 May 2015: New parameter genericRecordType to cater for records with no key field value as shown in scenario 3. Previous scenario 3 renamed to Scenario 4.

Update 25 Mar 2015: DeepFCCBean has been refactored to be part of FormatConversionBean. Parameter conversionType replaced with converterClass. Note that some of the screenshots are not updated to reflect the new parameter converterClass.

Update 17 Mar 2015: Parameter indentXML replaced by indentFactor.

Introduction

This is the second part of the two-part series on achieving file content conversion (FCC) for deep structure with a reusable custom adapter module. The first part covered the conversion of deep XML to flat file at a receiver channel.

This second part will focus on conversion of flat file to deep XML at a sender channel.

Source Code

Refer to following blog on location of source code and/or EAR deployment file.

FormatConversionBean - One Bean to rule them all!

Prerequisite

• Each substructure type must contain a field containing the key value to enable the parser to identify lines of this substructure type


• Each substructure type must belong to one and only one parent substructure type. The parent can be either the root element or other substructure types defined

Usage of Module in Communication Channel

Module Processing Sequence

Number	Module Name	Type	Module Key
<Depending on position of module in chain> 1) Asynchronous scenario, Normally before the last module in channels 2) Synchronous scenario, Before last module to convert request payload After last module to convert response payload	Custom_AF_Modules/FormatConversionBean	Local Enterprise Bean	<Any Arbitrary Value>

Module Parameter Reference

Below are the parameters for configuration of the module for flat file to deep XML conversion (converterClass = 'com.equalize.xpi.af.modules.deepfcc.DeepPlain2XMLConverter'). Certain parameters will automatically inherit the default values if it is not configured.

The parameters are separated into parameters that are common for this conversion type, and also parameters that are specific for field separator based or fixed length based conversion.

1) Common Parameters

Available only in PI

Parameter Name	Allowed values	Default value	Remarks
converterClass	PI - com.equalize.xpi.af.modules.deepfcc.DeepPlain2XMLConverter CPI - com.equalize.converter.core.DeepPlain2XMLConverter		Required field. Determines conversion class
encoding		UTF-8	Defines encoding of output plain text file
documentName			Required field. Document name of root element of XML output
documentNamespace			Required field. Namespace of root element of XML output
recordsetStructure			Required field. List of substructure names separated by comma
genericRecordType			Name of substructure used as generic type. If configured, any lines that cannot be determined via key field determination will be treated as this generic type.
keyFieldName			Required field. Name of the key field that must occur in all substructures
indentFactor	Integer values beginning from 1	0	Determines the number of indentation spaces for each level in the XML output
rowOffset	Integer values beginning from 1	0	Starting row to begin extracting content from (i.e. 0 = start from first row, 1 = start from second row)
trimContents	Y, N	Y	Determines if all the leading and subsequent blanks for a field will be trimmed or not
endSeparator			Defines default end separator for entire input file
<StructureName>.fieldNames			Required field. Names of fields in this substructure
<StructureName>.keyFieldValue			Required field. Key field value to identify this substructure
<StructureName>.parent			Required field. Specifies the parent of this substructure. Value should correspond to a substructure in recordsetStructure or the value 'Root' to indicate that it belongs directly under the root element
<StructureName>.missingLastFields	ignore,add,error	ignore	Behavior of conversion when there are less fields in content compared to configured fields in <StructureName>.fieldNames ignore = missing fields not included in output add = missing fields included with blank value in output error = conversion terminated with error due to incomplete input
<StructureName>.additionalLastFields	ignore,error	ignore	Behavior of conversion when there are more fields in content compared to configured fields in <StructureName>.fieldNames ignore = additional fields not included in output error = conversion terminated with error due to extraneous input
debug			Displays audit log messages of substructure and parent-child determination of each line. WARNING: Use this only for debugging in non-productive systems Available only in PI
messageLog	pre, post		Saves a log version of the message that is viewable in Message Monitor pre = saves payload before conversion post = saves payload after conversion Available only in PI
logLocation			Name of log version when messageLog is populated. Location defaulted to value in messageLog if logLocation not populated. Available only in PI

2) Field Separator Conversion Parameters

Parameter Name	Allowed values	Default value	Remarks
defaultFieldSeparator			Defines a default field separator for all substructures that do not have <StructureName>.fieldSeparator explicitly configured
<StructureName>.fieldSeparator			Either <StructureName>.fieldFixedLengths or <StructureName>.fieldSeparator must be populated
<StructureName>.enclosureSignBegin			Specify a string that acts as a text delimiter. Field separators within such texts are ignored
<StructureName>.enclosureSignEnd			If the text delimiters for the beginning and end of the text are different, specify the text delimiter for the end of the text here. If it is not populated, it uses value specified in <StructureName>.enclosureSignBegin
<StructureName>.enclosureSignBeginEscape			Specify a string that replaces the text delimiter if it occurs within a text that it delimits. When the text is transferred the string is replaced by the value specified in <StructureName>.enclosureSignBegin
<StructureName>.enclosureSignEndEscape			Specify a string that replaces the text delimiter for the end of the text if it occurs within a text that it delimits. When the text is transferred the string is replaced by the value specified in <StructureName>.enclosureSignEnd. If it is not populated, it uses value soecified in <StructureName>.enclosureSignBeginEscape
<StructureName>.enclosureConversion	Y, N	Y	Y = the text delimiter is to be removed and/or the escape character is to be replaced when the files are transferred N = the character is to be transferred unchanged

3) Fixed Length Conversion Parameters

Parameter Name	Allowed values	Default value	Remarks
<StructureName>.fieldFixedLengths	Integer separated by commas		Either <StructureName>.fieldFixedLengths or <StructureName>.fieldSeparator must be populated. The number of lengths must match the number of fields listed in <StructureName>.fieldNames

Similar to standard FCC functionality, fieldSeparator parameter supports non-printable ASCII characters as follows:

• characters encoded as hexadecimal values in the form of '0xHH' (including quotation marks)

Example Scenarios

Here are some example scenarios of the behavior of the conversion based on different configuration options.

Scenario 1

3 substructures with 3 level deep structure

Field separator based conversion

Field separator - Order and Item using default comma. Delivery using tab (specified in hexadecimal)

Order uses " as enclosure sign, and "" as enclosure escape

Item uses " as enclosure sign but enclosure sign is not removed when transferred

Row offset provided to skip first line

Output XML indented

Module parameters

Parameter Name	Parameter Value
converterClass	com.equalize.xpi.af.modules.deepfcc.DeepPlain2XMLConverter
documentName	MT_DeepPlain2XML
documentNamespace	urn:equalize:com
recordsetStructure	Delivery,Order,Item
keyFieldName	Type
Delivery.fieldNames	Type,DeliveryNo
Order.fieldNames	Type,DeliveryNo,OrderNo
Item.fieldNames	Type,OrderNo,ItemNo,Quantity
Delivery.keyFieldValue	D
Order.keyFieldValue	O
Item.keyFieldValue	I
Delivery.parent	Root
Order.parent	Delivery
Item.parent	Order
Delivery.fieldSeparator	'0x09'
defaultFieldSeparator	,
Order.enclosureSignBegin	"
Order.enclosureSignBeginEscape	""
Item.enclosureConversion	N
Item.enclosureSignBegin	"
rowOffset	1
indentFactor	4

Result

Input
Output

Scenario 2

4 substructures with 3 level deep structure

Fixed length based conversion

Leading and trailing whitespaces are not trimmed

Missing fields are automatically populated with blank

Output XML indented

Module parameters

Parameter Name	Parameter Value
converterClass	com.equalize.xpi.af.modules.deepfcc.DeepPlain2XMLConverter
documentName	MT_DeepPlain2XML
documentNamespace	urn:equalize:com
recordsetStructure	Header,Delivery,Order,Item
keyFieldName	Type
Header.fieldNames	Type,FileName
Delivery.fieldNames	Type,DeliveryNo
Order.fieldNames	Type,DeliveryNo,OrderNo
Item.fieldNames	Type,OrderNo,ItemNo,Quantity
Header.keyFieldValue	H
Delivery.keyFieldValue	D
Order.keyFieldValue	O
Item.keyFieldValue	I
Header.parent	Root
Delivery.parent	Root
Order.parent	Delivery
Item.parent	Order
Header.fieldFixedLengths	5,20
Delivery.fieldFixedLengths	5,10
Order.fieldFixedLengths	5,10,10
Item.fieldFixedLengths	5,10,10,10
trimContents	N
indentFactor	4

Result

Input
Output

Scenario 3

Conversion where one and only one substructure type does not have key field to identify the type of record

Substructure Detail is configured as the generic record type

Sample test data for this scenario is from the following thread File Content Conversion - No Key Field in Detail Structure

Module parameters

Parameter Name	Parameter Value
converterClass	com.equalize.xpi.af.modules.deepfcc.DeepPlain2XMLConverter
documentName	MT_DeepPlain2XML
documentNamespace	urn:equalize:com
recordsetStructure	Header,Detail,Footer
genericRecordType	Detail
keyFieldName	Type
Header.fieldNames	Type,HeaderField
Detail.fieldNames	Field1,Field2,Field3,Field4,Field5
Footer.fieldNames	Type,TrailerField
Header.keyFieldValue	HDR
Footer.keyFieldValue	TRL
Header.parent	Root
Detail.parent	Root
Footer.parent	Root
Header.fieldFixedLengths	3,11
Detail.fieldSeparator	,
Footer.fieldFixedLengths	3,9
indentFactor	2

Result

Input	HDRXXXYYY12345 MM8100422373,M631530008  ,1530008,031615,17 MN8100424411,M631530008  ,1530008,031615,22 XY8100422373,M631530008  ,1530008,031615,17 TRLXXZZZ4321
Output

Scenario 4

Finally for the fourth scenario, we will use the same input file that was used in the XSLT approach. Actual configuration and testing screenshots are shown.

Default field separator for all substructures is used.

A log version of the message payload before conversion is saved.

Module configuration on an SFTP sender channel.



The 'plain' log version shows the flat file before conversion.



The converted payload is sent to a receiver proxy without further mapping. As shown below, the target payload has been converted to a deeply nested XML structure.



The audit log shows the trace of steps being executed by the module.



Conclusion

With the completion of this second part for DeepFCCBean, we now have a generic, reusable and highly configurable custom adapter module that is able to handle conversion of flat file to deep XML and vice versa!