SAP Cloud Integration – Fixed Length Data Conversion for Full-Width Characters

[akirwsky]

Dear Reader,

This blog describes how to implement fixed length data conversion for full width characters such as Chinese and Japanese. Full-width characters are contained in two bytes and occupy a full-width square. There are some resources explaining how to convert characters with fixed length. And most of them are using XSLT Mapping. This solution did not work for my scenario with a mix of half and full-width chracters, and I realized this solution using XSLT mapping works fine when the data contains ONLY half-width characters. Therefore, I built my solution which works well with BOTH half-width and full-width characters. If you have any questions or suggestions to improve my solution, please kindly write in the comment. Thank you for reading in advance.

Scenario:

In my scenario, data should be sent to the target server with fixed length. And the data contains both half-width characters and full-width characters (Japanese Hiragana and Kanji). To explain why I had to use my solution, let’s compare the output data generated from the XSLT mapping solution with the one generated from my solution.

Field names and the fixed lengths.

	Field Name	Fixed Length (in bytes)
1	Number	8
2	Type	2
3	Date	8
4	Text	30
5	PIC	10
6	Location	6

Sample data

<Table>

<Record>

<Number>00000001</Number>

<Type>A</Type>

<Date>20230701</Date>

<Text>あいうえお</Text>

<PIC>Tim</PIC>

<Location>123456</Location>

</Record>

<Record>

<Number>00000002</Number>

<Type>B</Type>

<Date>20230701</Date>

<Text>漢字-FullWidth</Text>

<PIC>Lisa</PIC>

<Location>123456</Location>

</Record>

<Record>

<Number>00000003</Number>

<Type>C</Type>

<Date>20230701</Date>

<Text>FullWidthひらがな</Text>

<PIC>Mike</PIC>

<Location>123456</Location>

</Record>

   :

</Table>

 

Failure Case: XSLT Mapping

 

1. Input sample data

By using the content modifier, input the sample data.

 

2. XSLT mapping

You can refer to this resource how to create XSLT mapping file.

(Fixed Length File Generation Scenario Through SAP Cloud Platform Integration | SAP Blogs)

<?xml version="1.0" encoding="UTF-8"?>                                            

<xsl:stylesheet  xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">                                            

<xsl:output omit-xml-declaration="yes" indent="yes" method="text"/>                                            

<xsl:strip-space elements="*"/>                                            



<xsl:template match="Record">                                            

<xsl:apply-templates />                                            

<xsl:text>                                            

</xsl:text>                                            

</xsl:template>



<xsl:template match="Number">                                            

<xsl:value-of                                            

      select="substring(concat(., '        '), 1, 8)"/>                                            

</xsl:template>      

<xsl:template match="Type">                                            

<xsl:value-of                                            

      select="substring(concat(., '  '), 1, 2)"/>                                            

</xsl:template>                                            

<xsl:template match="Date">                                            

<xsl:value-of                                            

      select="substring(concat(., '        '), 1, 8)"/>                                            

</xsl:template>                                            

<xsl:template match="Text">                                            

<xsl:value-of                                            

      select="substring(concat(., '                              '), 1, 30)"/> 

</xsl:template>                                            

<xsl:template match="PIC">                                            

<xsl:value-of                                            

      select="substring(concat(., '          '), 1, 10)"/>                                            

</xsl:template>

<xsl:template match="Location">                                            

<xsl:value-of                                            

      select="substring(concat(., '      '), 1, 6)"/>                                            

</xsl:template>

</xsl:stylesheet>

I tried with other character codes such as Shift-JIS and MS932.

Shift-JIS

<?xml version="1.0" encoding="Shift-JIS"?> 

MS932

<?xml version="1.0" encoding="MS932"?> 

 

3. Check the output data

As you can see, the first characters of each PIC field’s value are not aligned exactly. I used Sakura Editor to make it easy to count the bytes of characters. -> Releases · sakura-editor/sakura (github.com)

UTF-8

Shift-JIS

MS932

 

Successful Case:

This solution uses three content modifiers and a groovy script. These three content modifiers are for passing parameters to the groovy script. If you want to manage the parameters in the script, you can remove these three content modifiers. I implemented in this way with three content modifiers so that anyone who don't know how to code can just copy and paste my code.

 

1. Input sample data.

(Same as the previous way)

2. Set fields names in the properties, using content modifier.

You need to use the same field names and seuquence of the fields as the ones in the input data.

 

3. Set fields fixed length in the properties, using content modifier.

For each field, type fixed length. And use the same seuquence of the fields as the one in the previous step. For example, the 'length1' should be the length of the 'field1' property in the previous content modifier. Make sure to use the fixed length in bytes, but not in characters. Again, even though one half-width character and one full-width can be considered as one character, they have different length in bytes. One half-width character is one byte, but one full-width character is two bytes. Therefore, you always need to convert chracters with fixed length in the unit of bytes.

 

4. Set other parameters such as the record name and the number of fields.

Set a closure name of the record and the number of fields. In this demonstration, because the sample data has '<Record></Record>' for the closure name of the records, type 'Record' in the source value of 'RecordName'. And because we have 6 fields, type 6 in the source value of 'Number OfFields'.

 

5. Write groovy script to convert characters with fixed length.

Take all the headers and properties we prepared in the previous steps and use those to iterate over the records and fields properly. This code will pad each field value with spaces to convert it with the fixed length in the unit of bytes.

import com.sap.gateway.ip.core.customdev.util.Message;

import java.util.HashMap;

import java.lang.Math;



def Message processData(Message message) {

    //take a message body, properties, headers

    def body       = message.getBody(java.lang.String) as String;

    def properties = message.getProperties();

    def headers    = message.getHeaders();

    

    //Get the number of records

    String record_name        = headers.get("RecordName")

    String[] lines_record     = body.split("<"+record_name+">", -1);

    int num_records           = lines_record.length;

    

    //Get the number of fields

    int num_fields = Integer.parseInt(headers.get("NumberOfFields"));



    //you will send this new message body

    def new_body   = new StringBuffer();

    

    //iterate over records

    for(int r=1; r<num_records; r++){

        //iterate over fields

        for(int f=1; f<=num_fields; f++){

            

            //take the field value

            String field_name      = properties.get("field"+String.valueOf(f));

            String[] lines1        = lines_record[r].split("<"+field_name+">", -1);

            String[] lines2        = lines1[1].split("</"+field_name+">", 2);

            String field_value     = lines2[0];

            

            //take the fixed length for the field

            int    fixed_length    = Integer.parseInt(properties.get("length"+String.valueOf(f)));

            

            //you will keep the padded field value here

            String padded;

            

            //calculate the number of spaces to pad

            //you need to choose a right character code for your language. Ex) Japanese -> Shift_JIS

            int bytes           = field_value.getBytes("Shift_JIS").length;

            int num_to_pad      = fixed_length - bytes;

            

            //if padding is necessary for the fixed length

            if (num_to_pad > 0){

                //pad the field value with spaces

                padded = field_value + new String(new char[num_to_pad]).replace('\0', ' ');

            }

     

            //case when bytes of the field value exceeds the fixed legnth

            else {

                

                //take current bytes and a current length of the field_value

                int cur_bytes = bytes;

                int cur_len   = field_value.length();

                

                //trim the last exceeding characters one by one

                while (cur_bytes > fixed_length){

                    cur_len     = field_value.length();

                    field_value = field_value.substring(0, cur_len-1);

                    cur_bytes   = field_value.getBytes("Shift_JIS").length;

                }

                

                //if the last trimmed character was full-width, you need to pad it with one more space

                if ( cur_bytes != fixed_length)

                    field_value = field_value + new String(new char[1]).replace('\0', ' ');

                

                //set in the padded

                padded = field_value;  

            }

            

            //append in the new_body

            new_body.append(padded);

        }

        

        //append a line break (CR/LF)

        new_body.append('\r\n');

    }



    //set the new body back to the message body

    message.setBody(new_body);

    return message;

}

 

6. Check output file.

As you can see, the first characters of each PIC fields and of Location fields are aligned beautifully.