Problem with loading special language characters from CSV files

[Former Member]

Hello,

I would like to load a CSV file with Data Services into an Oracle database table.

The CSV file contains special language characters (Umlaute like ä, ö, ü) as the conten is German.

But the special characters are not properly loaded to the oracle table.

Does anyone know a solution for this - in my opinion - standard task?

Which coding must the csv file have, UTF-8 or ANSI?

Thanks for your help in advance!

Philipp

[vnovozhilov]

Hi Philipp,

Please read the following article: http://wiki.sdn.sap.com/wiki/display/EIM/Multiple+Codepages

Thank you,

Viacheslav.

[Former Member]

Hi, Philipp.

As Michael stated, Data Services can handle any character set. However, this subject is tricky. There are many factors involved to assure you're reading the data and loading it properly, and also using the correct tools to check it. In the Designer, as your source file has special characters like ä, ö and ü I suggest you to set the code page to UTF-8 in the file definition.

In order to get the correct data loaded into the database, the first step is to verify what is the character set defined for that DB. This can be done by running the following SQL statement:

SELECT * FROM nls_database_parameters WHERE parameter = 'NLS_CHARACTERSET';

The returned character set should be set in the datastore properties for the 'Server code page' and 'Code pager' parameters. 'Server code page' defines what character set the Job Server (JS) will use to write the data into the DB, 'Code page' defines what character set Designer will use to show you the data when you click the magnifying glass of the target table. There is one more important thing to check if your JS is running in a Linux/Unix machine: the value of the $NLS_LANG environment variable. This environment variable is used by the Oracle client to write the data into the DB. It must be set accordingly with the value set in the Datastore properties ('Server code page'). This done should be enough to get the data written properly.

Note: when checking if the data was correctly written into the DB, be sure to check the character set that is used by the application to show you the data. Sometimes the correct data is written in the DB but the query tool is configured to read it using another character set so you can be deceived.

I hope this helps you.

Regards,

Leonardo.

[former_member718193]

Thanks Leanardo for solution. This also helped in solving our problem.

[Former Member]

Hello

Data Services will handle any character set, but it is up to the developer to set the right codepage and language.  File formats, and many datastores allow you to specify these locale settings.  I suspect you have might have something set to "default".

If I ever have an issue loading characters I set everything to UTF-8 of UTF-16.  Your Oracle database must also be configured to handle the characters too.  AL32UTF8 is the mother of all character sets.

Michael