topic Re: Error during file read from Application Server -Unicode tab Delimited Text file in Application Development and Automation Discussions

Error during file read from Application Server -Unicode tab Delimited Text file

Former Member — Wed, 18 May 2016 02:59:40 GMT

Hi we have a requirement to read a file of type Unicode Tab delimited file ( containing chinese characters) from Application server.

i have tried in different ways , but i face following issue.

Out System is Uncode and the file is prepared in Excel and saved as " Unicode( tab delimited) Text file".

1. I tried with OPEN DATASET p_inp_file FOR INPUT IN TEXT MODE ENCODING default. but it failed at statament READ DATASET p_inp_file INTO lwa_line . statement with error "CX_SY_CONVERSION_CODEPAGE"

What happened?

During conversion of a text from code page '4110' to code page '4102',

one of the following occurred:

OPEN DATASET p_inp_file FOR INPUT IN LEGACY TEXT MODE

BIG ENDIAN CODE PAGE '4102'. Got below error immediately after executing the open data set statement.

Error:

The conversion of certain code pages is not supported.

What happened?

The conversion of texts in code page '4102' to code page '4102' is not

supported here.

OPEN DATASET p_inp_file FOR INPUT IN LEGACY TEXT MODE

BIG ENDIAN CODE PAGE '4110'. Got below error immediately after executing the open data set statement.

Error:

The conversion of certain code pages is not supported.

What happened?

The conversion of texts in code page '4110' to code page '4102' is not

supported here.

OPEN DATASET p_inp_file FOR INPUT IN LEGACY TEXT MODE BIG ENDIAN CODE PAGE '1100'

IGNORING CONVERSION ERRORS . I see that a special charecter '#' is added after each charetcre of the input file.

ex: if one of the field value is : Z1MAT , after reading the file it was:#Z#1#M#A#T

read1.PNG

from the front end(PC) im able to read the same file with out any conversion, but same file when transferred to application server not working.

Please help to suggest how to read this Unicode tab Delimited Text file from application server. I see that its current code page 4102.

Thanks

Re: Error during file read from Application Server -Unicode tab Delimited Text file

pranay570708 — Wed, 18 May 2016 05:06:05 GMT

Hi,

Try this:

open dataset P_FILE in text mode encoding default for input ignoring conversion errors.

Re: Error during file read from Application Server -Unicode tab Delimited Text file

Former Member — Wed, 18 May 2016 05:17:26 GMT

I tried that also, result is same as in case(4) above:

. I see that a special charecter '#' is added after each charetcre of the input file.

ex: if one of the field value is : Z1MAT , after reading the file it was:#Z#1#M#A#T

Thanks

Re: Error during file read from Application Server -Unicode tab Delimited Text file

pranay570708 — Wed, 18 May 2016 05:31:01 GMT

Use below code to remove '#'.

REPLACE ALL OCCURRENCES OF '#' IN LWA_LINE WITH SPACE.

CONDENSE LWA_LINE.

Re: Error during file read from Application Server -Unicode tab Delimited Text file

Former Member — Wed, 18 May 2016 05:49:32 GMT

1.Beforr replace statement

2.After Replace statment

No change. Looks like its not hash but some code page related character?

Pasted the Text from above debug screen:

A#g#r#e#e#m#e#n#t# #T#y#p#e###P#r#o#j#e#c#t# #C#o#d#e###S#a#l#e#s# #O#r#g###D#i#s#t#C#h#a#n###D#i#v###C#u#s#t#g#r#p###C#h#a#n#n#e#l#

Thanks

Re: Error during file read from Application Server -Unicode tab Delimited Text file

Sandra_Rossi — Wed, 18 May 2016 06:27:00 GMT

Excel "unicode text" (tab delimited) format is saved in UTF-16 Little Endian (with a BOM "FFFE" in the first 2 bytes).

There can be an issue at 2 moments: either during the upload of the file from the laptop to the application server, or when you read the file from the application server.

You must be sure which character encoding you upload to the application server. I advise to upload using BIN mode so that there's no conversion, and so you should have a file on the application server in UTF-16 little endian.

SAP code page for UTF-16 little endian (UTF-16LE) is "4103".

EDIT 22/08/2022:

OPEN DATASET cannot read directly UTF-16 (one of the only code page with that issue). The solution is to read directly in binary mode, and then convert.

DATA xstring TYPE xstring.
OPEN DATASET p_inp_file FOR INPUT IN BYTE MODE.
READ DATASET p_inp_file INTO xstring.
CLOSE DATASET p_inp_file.
DATA(string) = cl_abap_codepage=>convert_from( source = xstring codepage = `UTF-16LE` ).
IF string IS NOT INITIAL.
  string = string+1. " remove BOM character
ENDIF.

Re: Error during file read from Application Server -Unicode tab Delimited Text file

Former Member — Wed, 18 May 2016 06:35:15 GMT

Hi Sandra Rossi, Thanks

I tried,

OPEN DATASET p_inp_file FOR INPUT IN LEGACY TEXT MODE

CODE PAGE '4103' message lv_mess.

i got the below dump immediately after the above statement.

The conversion of texts in code page '4103' to code page '4102' is not

supported here.

I have already checked that when using gui_upload the code page 4103 is used, but on application server the code page is 4102.

Im using CG3Z transaction Binary mode to transfer the file TO appication server.

Re: Error during file read from Application Server -Unicode tab Delimited Text file

Sandra_Rossi — Wed, 18 May 2016 07:18:19 GMT

Sorry, that's true, as I could read in that post: "Unfortunately, the OPEN DATASET statement does not support UTF-16. A workaround is to open the file in BINARY MODE and to use the classes CL_ABAP_CONV_IN_CE and CL_ABAP_CONV_OUT_CE."

Re: Error during file read from Application Server -Unicode tab Delimited Text file

pranay570708 — Wed, 18 May 2016 08:58:17 GMT

Pls give it a try to remove special character '#'.

CONSTANTS: C_HASH TYPE C VALUE CL_ABAP_CHAR_UTILITIES=>HORIZONTAL_TAB.

REPLACE ALL OCCURRENCES OF C_HASH IN LWA_LINE WITH SPACE.

CONDENSE LWA_LINE.

Re: Error during file read from Application Server -Unicode tab Delimited Text file

Sandra_Rossi — Wed, 18 May 2016 12:24:00 GMT

example of code:

DATA: l_string TYPE string,
      l_xstring TYPE xstring,
      lo_conv TYPE REF TO cl_abap_conv_in_ce,
      lt_line TYPE TABLE OF string.

OPEN DATASET p_inp_file FOR INPUT IN BINARY MODE.
READ DATASET p_inp_file INTO l_xstring.
CLOSE DATASET p_inp_file.

lo_conv = cl_abap_conv_in_ce=>create( encoding = '4103' input = l_xstring ).
lo_conv->skip_x( n = 2 ). " skip BOM
lo_conv->read( IMPORTING data = l_string ).
SPLIT l_string AT cl_abap_char_utilities=>cr_lf INTO TABLE lt_line.

(Written directly in the forum, not tested)

Re: Error during file read from Application Server -Unicode tab Delimited Text file

sh4il3sh — Mon, 22 Aug 2022 11:14:47 GMT

Hi sandra.rossi , could you please explain the significance of Skip BOM line?
3rd last line from bottom.

Re: Error during file read from Application Server -Unicode tab Delimited Text file

Sandra_Rossi — Mon, 22 Aug 2022 12:43:07 GMT

sh4il3sh

I edited my old answer because it was incorrect.

Re: Error during file read from Application Server -Unicode tab Delimited Text file

sh4il3sh — Mon, 22 Aug 2022 13:52:45 GMT

Thank You sandra.rossi .
I just found that below statement works like charm, it was added in the release 6.40.

OPEN DATASET filename FOR INPUT IN TEXT MODE ENCODING UTF-8 SKIPPING BYTE-ORDER MARK WITH SMART LINEFEED IGNORING CONVERSION ERRORS.

Is there any way to determine if my internal table has special characters?
If there is no UTF file, I need to generate non utf-8 file in AL11 and vice versa.

Re: Error during file read from Application Server -Unicode tab Delimited Text file

Sandra_Rossi — Tue, 23 Aug 2022 08:20:47 GMT

sh4il3sh

my answer was about UTF-16, not UTF-8.

Please ask a new question, and clarify which "special characters" and what kind of "non utf-8" file you are talking about.

NB: you should not say "a file in AL11", you should say "a file in the file system of the application server". AL11 is just to view the file system.