OPEN DATASET syntax for Unicode and non-Unicode, mixed data

[vihaan2018]

Hello Experts,

(1) Our ECC system is non-unicode system

We have a custom prog it is reading Acc. doc BKPF data and storing in App server, then other legacy non-SAP system is consuming the same data from App server, this prog has below syntax

OPEN DATASET <file name> FOR OUTPUT IN TEXT MODE MESSAGE <my messsages>

ENCODING DEFAULT

TRANSFER <my rec> TO <file name>

The above is worked fine

(2) Recently we unicode enabled our ECC system,

Then the receiving / consuming non-SAP legacy system is throwing an error as "Truncation of a record occurred during I/O operation"

After investigation we saw that the NBSP (Non blank space in Notepad++) in BKPF-BKTXT is causing the issue of truncation then we  changed the ABAP syntax to as below

OPEN DATASET <file name> FOR OUTPUT IN TEXT MODE MESSAGE <my messsages>

ENCODING NON-UNICODE

Worked fine, no longer NBSP issue

(3) But yesterday we got dump in SAP as CONVT_CODEPAGE with below message

"When converting a text from code page '4102' to code page '1100', either
- characters are detected that cannot be displayed in one of the two code pages or
- the conversion could not be carried out for another reason"

(4) So it means that the BKPF-BKTXT (free/long text) has non-unicode for some acc. docs and unicode text for some other acc. docs

Pls. let us know a syntax that works for non-unicode text and unicode text, can we go with BINARY format like below,

OPEN DATASET <fine name> FOR OUTPUT IN BINARY MODE

[Sandra_Rossi]

Please don't post to Questions about SAP Websites (issue with community Website, etc.) if it's a question about SAP software. Instead, ask in Technology Q&A: https://community.sap.com/t5/technology-q-a/qa-p/technology-questions.

Hopefully someone will fix that for you.

[Sandra_Rossi]

Not related to the error, but you should indicate explicitly which code page is expected by the receiving system, not NON-UNICODE which varies depending on the linguistic environment as explained in the ABAP documentation of OPEN DATASET.

[Sandra_Rossi]

Conversion of characters from Unicode to non-Unicode will potentially lead to conversion errors -> https://help.sap.com/doc/abapdocu_latest_index_htm/latest/en-US/index.htm?file=abapopen_dataset_erro...

[vihaan2018]

@Rossi, Thank you for the suggestions, you mean, I need to write the syntax as below, just for example purpose?

[vihaan2018]

 @Sandra_Rossi Thank you for the suggestions, you mean, I need to write the syntax as below, just for example purpose?

OPEN DATASET <file name> FOR OUTPUT IN TEXT MODE MESSAGE <my messages> ENCODING '1100'

[raymond_giuseppi]

Not 'non blank space' but 'non-breaking space '

[Sandra_Rossi]

In my last comment, I meant that if the SAP code page '1100' (Latin 1 / ISO-8859-1) is expected by the receiving system, you should use OPEN DATASET ... CODE PAGE '1100' ... explicitly, not NON-UNICODE which varies depending on the linguistic environment as explained in the ABAP documentation of OPEN DATASET. I let you find the right syntax to use CODE PAGE '1100' in the ABAP documentation.

[nils_buerckel]

Hi,

this is a general problem when exchanging char-like data between Non-Unicode and Unicode based systems. It can happen during transport, interfacing (e.g. via RFC) or (as in your case) file exchange.

 Unicode is supporting more than 100000 characters, while code page 1100 (ISO-Latin-1) supports less than 256. As soon as a character outside the Non-Unicode codepage is transferred, the system needs raise an error or to replace it with a replacement character (default in SAP is "#") Alternatively it might be also possible (as already mentioned) to convert the problematic character via transliteration.

However when transferring textual data from Unicode to Non-Unicode, you cannot avoid risk of data loss - this will always be recognized (at the latest) in the receiving Non-Unicode system.

By the way using binary mode is not an option at all - the content will not be readable anymore ...

The best solution would be to use Unicode systems in the landscape only, but this is often not possible in reality ....

Please also have a look at SAP note 1322715 and the attachment of that SAP note called  Requirements_of_ABAP_Programs_in_Unicode_Systems.pdf.

Best regards,

Nils 

[nils_buerckel]

Unicode

[nils_buerckel]

Unicode

[vihaan2018]

@raymond_giuseppi yes, you are correct, thank you. Do you have any thoughts on our issue because the receiving system analyst said their code page is IBM 37 (CCSID37) with full Latin-1 char set 

[raymond_giuseppi]

Don't answer as an answer to oriinal question, use comment on the previous answer/comment

[raymond_giuseppi]

Consider using REPLACEMENT CHARACTER rc and IGNORING CONVERSION ERRORS in the OPEN DATASET statement to handle Unicode character that cannot be converted to your non Unicode page code (and are not handled thru transliteration. Look for FM such as SCP_REPLACE_STRANGE_CHARS)

As your users are now able to copy/paste exotic character to SAP from Word or any Web sources, this is likely to become a frequent occurrence.  

NB: Does you company use some ESB, those tools are often able to perform those conversions.

[raymond_giuseppi]

IBM 37 is a Latin1 EBCDIC page code.

• Try to replace non-suitable character with FM SCP_REPLACE_STRANGE_CHAR in extracted data (text fields)
• Then add the already suggested options to the OPEN DATASET statement

Be aware that SCP_REPLACE_STRANGE_CHAR can and will change number of characters (e.g. '±' > '+/-' or '€' > 'EUR'

[vihaan2018]

@Sandra_Rossi Thank you for the suggestions, you mean, I need to write the syntax as below, just for example purpose?

OPEN DATASET <file name> FOR OUTPUT IN TEXT MODE MESSAGE <my message> ENCODING '1100'

[Sandra_Rossi]

Please don't post an answer, which is reserved to propose a solution. Instead click on "Show replies" and then "Comment".

[Sandra_Rossi]

This solution was marked as an "accepted solution" but it leads to a syntax error/doesn't compile, so I revert the solution flag. Please don't mislead other people. Thank you.