Error during file read from Application Server -Unicode tab Delimited Text file

[Former Member]

Hi we have a requirement to read a file of type Unicode Tab delimited file ( containing chinese characters) from Application server.

i have tried in different ways , but i face following issue.

Out System is Uncode and the file is prepared in Excel and saved as " Unicode( tab delimited) Text file".

1.   I tried with OPEN DATASET p_inp_file FOR INPUT IN TEXT MODE ENCODING  default. but  it failed at statament READ DATASET p_inp_file INTO  lwa_line  . statement with error "CX_SY_CONVERSION_CODEPAGE"

What happened?

    During conversion of a text from code page '4110' to code page '4102',

    one of the following occurred:

2.

OPEN DATASET p_inp_file  FOR INPUT IN LEGACY TEXT MODE

    BIG ENDIAN CODE PAGE '4102'. Got below error immediately after executing the open data set statement.

Error:

The conversion of certain code pages is not supported.

What happened?

    The conversion of texts in code page '4102' to code page '4102' is not

    supported here.

3.

OPEN DATASET p_inp_file  FOR INPUT IN LEGACY TEXT MODE

    BIG ENDIAN CODE PAGE '4110'. Got below error immediately after executing the open data set statement.

Error:

The conversion of certain code pages is not supported.

What happened?

The conversion of texts in code page '4110' to code page '4102' is not

supported here.

4.

OPEN DATASET  p_inp_file FOR INPUT IN LEGACY  TEXT MODE BIG ENDIAN CODE PAGE '1100'

                          IGNORING CONVERSION ERRORS . I see that a special charecter '#' is added after each charetcre of the input file.

ex: if one of the field value is : Z1MAT , after reading the file it was:#Z#1#M#A#T

read1.PNG

from the front end(PC) im able to read the same file with out any conversion, but same file when transferred to application server not working.

Please help to suggest how to read this Unicode tab Delimited Text file from application server. I see that its current code page 4102.

Thanks

[Former Member]

Sorry, that's true, as I could read in that post: "Unfortunately, the OPEN DATASET statement does not support UTF-16. A workaround is to open the file in BINARY MODE and to use the classes CL_ABAP_CONV_IN_CE and CL_ABAP_CONV_OUT_CE."

[pranay570708]

Hi,

Try this:

open dataset P_FILE in text mode encoding default for input ignoring conversion errors.

[Former Member]

I tried that also, result is same as in case(4) above:

. I see that a special charecter '#' is added after each charetcre of the input file.

ex: if one of the field value is : Z1MAT , after reading the file it was:#Z#1#M#A#T

Thanks

[pranay570708]

Use below code to remove '#'.

REPLACE ALL OCCURRENCES OF '#' IN LWA_LINE WITH SPACE.

CONDENSE LWA_LINE.

[Former Member]

1.Beforr replace statement

2.After Replace statment

No change. Looks like its not hash but some code page related character?

Pasted the Text from above debug screen:

A#g#r#e#e#m#e#n#t# #T#y#p#e###P#r#o#j#e#c#t# #C#o#d#e###S#a#l#e#s# #O#r#g###D#i#s#t#C#h#a#n###D#i#v###C#u#s#t#g#r#p###C#h#a#n#n#e#l#

Thanks

[pranay570708]

Pls give it a try to remove special character '#'.

CONSTANTS: C_HASH  TYPE C VALUE CL_ABAP_CHAR_UTILITIES=>HORIZONTAL_TAB.

REPLACE ALL OCCURRENCES OF C_HASH IN LWA_LINE WITH SPACE.

CONDENSE LWA_LINE.

[Sandra_Rossi]

Excel "unicode text" (tab delimited) format is saved in UTF-16 Little Endian (with a BOM "FFFE" in the first 2 bytes).

There can be an issue at 2 moments: either during the upload of the file from the laptop to the application server, or when you read the file from the application server.

You must be sure which character encoding you upload to the application server. I advise to upload using BIN mode so that there's no conversion, and so you should have a file on the application server in UTF-16 little endian.

SAP code page for UTF-16 little endian (UTF-16LE) is "4103".

EDIT 22/08/2022:

OPEN DATASET cannot read directly UTF-16 (one of the only code page with that issue). The solution is to read directly in binary mode, and then convert.

DATA xstring TYPE xstring.
OPEN DATASET p_inp_file FOR INPUT IN BYTE MODE.
READ DATASET p_inp_file INTO xstring.
CLOSE DATASET p_inp_file.
DATA(string) = cl_abap_codepage=>convert_from( source = xstring codepage = `UTF-16LE` ).
IF string IS NOT INITIAL.
  string = string+1. " remove BOM character
ENDIF.

[Former Member]

Hi Sandra Rossi, Thanks

I tried,

  OPEN DATASET p_inp_file FOR INPUT IN LEGACY TEXT MODE

  CODE PAGE '4103' message lv_mess.

i got the below dump immediately after the above statement.

The conversion of texts in code page '4103' to code page '4102' is not

supported here.

I have already checked that when using gui_upload the code page 4103 is used, but on application server the code page is 4102.

Im using CG3Z transaction Binary mode to transfer the file TO appication server.

[Sandra_Rossi]

Sorry, that's true, as I could read in that post: "Unfortunately, the OPEN DATASET statement does not support UTF-16. A workaround is to open the file in BINARY MODE and to use the classes CL_ABAP_CONV_IN_CE and CL_ABAP_CONV_OUT_CE."

[Sandra_Rossi]

example of code:

DATA: l_string TYPE string,
      l_xstring TYPE xstring,
      lo_conv TYPE REF TO cl_abap_conv_in_ce,
      lt_line TYPE TABLE OF string.

OPEN DATASET p_inp_file FOR INPUT IN BINARY MODE.
READ DATASET p_inp_file INTO l_xstring.
CLOSE DATASET p_inp_file.

lo_conv = cl_abap_conv_in_ce=>create( encoding = '4103' input = l_xstring ).
lo_conv->skip_x( n = 2 ). " skip BOM
lo_conv->read( IMPORTING data = l_string ).
SPLIT l_string AT cl_abap_char_utilities=>cr_lf INTO TABLE lt_line.

(Written directly in the forum, not tested)

[sh4il3sh]

Hi sandra.rossi , could you please explain the significance of Skip BOM line?
3rd last line from bottom.

[Sandra_Rossi]

sh4il3sh

I edited my old answer because it was incorrect.

[sh4il3sh]

Thank You sandra.rossi .
I just found that below statement works like charm, it was added in the release 6.40.

OPEN DATASET filename FOR INPUT IN TEXT MODE ENCODING UTF-8 SKIPPING BYTE-ORDER MARK WITH SMART LINEFEED IGNORING CONVERSION ERRORS.

Is there any way to determine if my internal table has special characters?
If there is no UTF file, I need to generate non utf-8 file in AL11 and vice versa.

[Sandra_Rossi]

sh4il3sh

my answer was about UTF-16, not UTF-8.

Please ask a new question, and clarify which "special characters" and what kind of "non utf-8" file you are talking about.

NB: you should not say "a file in AL11", you should say "a file in the file system of the application server". AL11 is just to view the file system.