Smart Data Integration - XML 2 Virtual Table Fails

[#1]

We are trying to acces a number of local Files via SMART DATA INTEGRATION. Local Files are in XML, and File can be up to 500MB. For this case the config file for the Remote Source has been set up as follows :

myXmlFile

FORMAT=XML

FORCE_FILENAME_PATTERN=myXMLFile.xml

PARTITIONS=0

CODEPAGE=UTF-8

LOCALE=en_US

ROOT_NAME=document

CIRCULAR_LEVEL=0

COLUMN=DATA;NCLOB;

First executions with comparable small number of records were working fine. And we can Extract Data via SQL Comand/Function XMLTABLE or can see the Content of Column DATA in the Hana Database Explorer RAW DATA - Preview. After scale up the number of records in the same file with the already included data ( just copied the content multiple times ) and reaching a file size above 23MB we get following error :

Could not open 'mySchema'.'myXMLFile'.

Error: (dberror) [7]: feature not supported: exception 70029042: wrong encoding found while read/write (N)CLOB: INPLACE_NCLOB: (at pos 73)

Files below this size working fine.

In the most cases you will assume that their is something wrong with the content. NOPE !
Same content was working with a small number of records. As i mentioned above, i just copied lines an pasted
them again in the same file.

Does someone have a clue what the reson could be for that behavior ?
Is the NCLOB Size limited ? Is there a Parameter to Increase that ?

We have tried a lot of Combination in the Config File with no Succes 😞

I appreciate all Help i could get.

Thanks in Advance.

Regards,

A.T

[jim_havlicek]

Hi A.T.

there are two configuration parameters that control the maximum size of XML or NCLOB data. These can also have the effect of truncating the data. Both of these need to be increased in order to load large XML or NCLOB files. For example, to load XML files up to 60MB in the presence of double-byte characters, apply the following settings.

1. On dpagent, in dpagentconfig.ini, framework.maxDataSize=120MB. The default is 10MB, but be careful to account for double-byte characters.
2. On Hana, ALTER SYSTEM ALTER CONFIGURATION ('indexserver.ini', 'SYSTEM') SET ('smart_data_access', 'max_buffer_length') = '125829120' WITH RECONFIGURE; Depending on your Hana version, this might default as low as 10MB.
3. While not directly related, you might have to increase the heapsize on the agent (in in dpagent.ini, -Xmx10240m). I suspect you aren't having that problem yet.

If this doesn't solve the problem, open a customer support ticket and mention my name - it'll find it's way to me.

Hope that helps,

Jim