CPI: Groovy - java.io.Reader eachLine is taking a long time

[suwandi_cahyadi]

Hi,

I have a script that will read the content of a CSV file line by line and do an operation for each line.

However the interface is taking a long time at this script step, it takes 24mins to finish processing a file with 30k lines of content:

So it actually causing a bottleneck in the whole process. This is the groovy script:

    def body = message.getBody(java.io.Reader);
    def xmlBody = ""
    
    // header
    xmlBody = '<?xml version="1.0" encoding="utf-8"?><ns:root>'

    // loop the file content line by line
    body.eachLine { line ->
            xmlBody += '<Records>'
            String[] data = line.split(',')
            // set each field
            xmlBody += '<field1>' + data[0].trim() + '</field1>'
            xmlBody += '<field2>' + data[1].trim() + '</field2>'
            xmlBody += '<field3>' + data[2].trim() + '</field3>'
            xmlBody += '</Records>'
        }
    }
    xmlBody += '</ns:root>'
    message.setBody(xmlBody);

Is there any other way to read a file content line by line in a faster way?

EDIT: Reason why I don't use the pre-defined CSV to XML converter is because it's not a simple conversion and additional logic is required (splitting 1 csv field to 2 xml elements)

Thank you.

[MortenWittrock]

Hi Suwandi

Because of String concatenation (gluing strings together with the + operator), the code creates many tens of thousands of String objects that get bigger and bigger for each iteration. That's why it's super slow.

Also, it's not a good idea to create XML by concatenating strings (with or without the + operator). An XML library would handle special characters for you, and generally make sure that the output is well formed XML.

Regards,

Morten

[Andrzej_Filusz · Accepted solution]

Hello,

Could you check the following code please?

def Message processData(Message message) {

    Reader reader = message.getBody(java.io.Reader)
    StringBuilder sb = new StringBuilder('<?xml version="1.0" encoding="utf-8"?><ns:root>')

    reader.eachLine { line ->
        String[] data = line.split(',')

        sb.append("<Records><field1>${data[0].trim()}</field1>")
        sb.append("<field2>${data[1].trim()}</field2>")
        sb.append("<field3>${data[2].trim()}</field3></Records>")
    }
    sb.append('</ns:root>')

    message.setBody(sb.toString())
    return message
}

Regards,

Andrzej

[suwandi_cahyadi · Accepted solution]

Thank you for the solution, I just knew String + operator is a costly operation when done many times.

[Yogananda]

suwandi.cahyadi

try it if this could work faster in loop

// loop the file content line by line
body.eachLine { line ->
xmlBody += '<Records>'
String[] data = line.split(',')
// set each field dynamically using loop
for (int i=0; i<data.length; i++) {
xmlBody += '<field' + (i+1) + '>' + data[i].trim() + '</field' + (i+1) + '>'
}
xmlBody += '</Records>'
}

[suwandi_cahyadi]

hi,

actually the <field1>,<field2>,<field3> those have been changed for privacy reason. In actual code they are not named that way.

[Yogananda]

suwandi.cahyadi

okay but you can slightly modify to your need in the code.