How to manage Unicode imports and exports in SAP Sourcing

[Former Member]

Prerequisites:

1. The database is set to support Unicode by having the correct character set specified during installation
2. The application server’s JVM is set to UTF-8 encoding
3. The used import files do support and are saved with UTF-8 encoding

Overview

In general multi-language support is very commonplace nowadays yet can pose some difficulty, depending on how the software has been setup and configured.

For SAP Sourcing this can happen if, for example, the import files are not properly prepared regarding file encoding.

Hence, after the setup that is performed by the System Administrator/Application Responsible, the end user solely has to pay attention to the typical import/export tasks he or she needs to perform.

Explanation of involved components

• System property “system.csv.export.encoding.use_default_encoding” that can be set on System Manager level
  
• - User Account setting to switch from text (CSV/TXT) to Excel and vice versa
  
• - Document Type setting to switch from text (CSV/TXT) to Excel and vice versa
  

Explanation and how to approach imports:

Once you`ve made sure that the DB supports Unicode (this should by default for a successful SAP Sourcing installation) you can continue to look at the application server. This seems like an area that`s somewhat overlooked as it`ll end up in a very different behaviour for the exact same workflow in the application.

Hence have a look at the below two tables depending on the application server`s OS character encoding (Cp1252 due to the fact that I was using a Windows based OS for this example)

File imports:
Application server JVM set to UTF-8 encoding:	Upload Preview	Trace content	Data in UI/DB
CSV	not OK	not OK	not OK
TXT	OK	OK	OK
Excel	OK	OK	OK
(all files UTF-8 encoded)
Application server JVM using default OS (Cp1252) encoding:	Upload Preview	Trace content	Data in UI/DB
CSV	not OK	not OK	not OK
TXT	OK	not OK	OK
Excel	not OK	not OK	not OK
(all files UTF-8 encoded)

It is important to note that that the Unicode “txt” might work if you are using the application server`s default encoding, but it will get confusing once you get to the trace. The preview and data will be correct, however, the trace might be confusing to users. Hence it is imperative to make sure that the application server is also set to use Unicode (UTF-8) encoding. From the start this will make it easier on the end user as they solely have to pay attention to the import file itself.

Explanation and how to approach exports:

If the data is already properly stored in Unicode in the database an export to e.g. an Excel file will be successful in its encoding.

To sum this up please have a look at the following table:

Application server JVM set to UTF-8 encoding:	Export file
CSV*	not OK
TXT**	OK
Excel	OK
* = Uncheck "Export to Excel" in the specific "User Account" settings
** = Set system property "system.csv.export.encoding.use_default_encoding" to "false"
Application server JVM using default OS (Cp1252) encoding:	Export file
CSV*	not OK
TXT**	OK
Excel	OK
* = Uncheck "Export to Excel" in the specific "User Account" settings
** = Set system property "system.csv.export.encoding.use_default_encoding" to "false"

Conclusion and recommendation

As a conclusion of this article it is highly recommended to make sure that the application server`s JVM is set to use UTF-8 encoding if you intend to use multiple languages.

On a NW AS this can be done  in the NW`s configtool by setting the Java JVM parameter “Dfile.encoding” with the value “UTF-8”.

If this is correctly set, the SAP Sourcing system will show in Setup – System Administration – System Information – Service Registration tab the “Character Set” which then would read UTF-8.

You may also read further in SAP KBA 2002598.

Let me know if you have any questions on this and/or also see my blog at: joerg.lippmann/blog for more details on SAP Sourcing.

Regards,

Joerg Lippmann

FAQs

1. Would you need a specific Sourcing language pack to be installed to support specific uploads?
   
   A: Actually no. The language pack mainly affects the SAP Sourcing standard page texts in the UI, not the user-entered data.
   
   
   
2. Would you need to have the system property “system.csv.export.encoding.use_default_encoding” set to support Unicode imports?
   
   A: No. You can take a Unicode (UTF-8) tab separated txt file and can directly import this into a SAP Sourcing system that has this property set (by default) to “true”.
   
   
   
3. Can I use Excel to save CSV files with Unicode character encoding?
   
   A: Not directly. In Excel, you can choose File à Save As… and save the file as type “Unicode Text (*.txt)”. Then rename the file extension from TXT to CSV. Excel will not save certain file types with Unicode encoding.
   
   
   
4. Why do I get “txt” files everywhere after setting “system.csv.export.encoding.use_default_encoding” to “false”?
   
   A: This is to make sure that the encoding is properly stored and supported. Even if the SAP Sourcing UI might still in certain places refer to CSV, all files that used to be of type CSV will be exported as TXT.