Symbols[Register, Tradermark, Copyright] are not loading in database via hot folder configuration

[Former Member]

Symbols[Register, Tradermark, Copyright] are not loading in database via hot folder configuration

Below are the configurations which i have done in my xml file:

And in the CSV I will be getting the values like below:

456,"High Availability™ Clusters and Options™" 789,"Test® Servers"

After reading the above csv via hotfolder from Linux OS we see the values in backoffice like below:

456 -> High Availability™ Clusters and Options™ 789 -> Test® Servers

Data should load as is from the CSV into Hybris DB.

Can any one please let me know what is wrong in my configuration.

Note:

It should load the data as is which is available[especially symbols] in any OS.

Unicode's through also i tried but no luck.

[Former Member]

link text

PFA the attached Hotfolder XML configurations

12141-xml-file.txt

[arvind-kumar_avinash]

I do not see any problem with the configuration that you have uploaded. The problem seems to be with the files which are being dropped into the hot folder i.e. they may not be UTF-8 encoded.

Just to understand what difference it can make, you can check an example like https://www.webtoffee.com/how-to-save-csv-excel-file-as-utf-8-encoded/

You can perform a simple test on your file using the UNIX command, file

 $ file your-file.csv

If it does not give you an output like your-file.csv: UTF-8 Unicode text, you need to make sure that the files being dropped into the Hot Folder are UTF-8 encoded. The best thing about this command is that it accepts wildcard characters e.g. $ file *.csv will show you the encoding of all the .csv files in the given folder.

In case you are using Windows, you can open your file using Notepad++ and check the Encoding menu to find its encoding.

If the problem still persists, please let me know the Hybris version and Database version to analyze the problem further.

[Former Member]

Arvind.,

Thanks a lot for your Swift response. I will check and get back to you.

[Former Member]

Files are coming in UTF-8 format only, but still we see the similar issue.

Is there any thing we need to apply UTF-8 for DB configurations at local.properties file ?

[Former Member]

Yes above one is there in my server. Here you go:

tomcat.generaloptions=-Xmx8096M -XX:-UseLoopPredicate -XX:MaxPermSize=1024M -XX:+HeapDumpOnOutOfMemoryError -ea -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dorg.tanukisoftware.wrapper.WrapperManager.mbean=true -Djava.endorsed.dirs="%CATALINA_HOME%/lib/endorsed" -Dcatalina.base=%CATALINA_BASE% -Dcatalina.home=%CATALINA_HOME% -Dfile.encoding=UTF-8 -Dlog4j.configuration=log4j_init_tomcat.properties -Djava.util.logging.config.file=jdk_logging.properties -Djava.io.tmpdir="${HYBRIS_TEMP_DIR}" -Djava.net.preferIPv6Addresses=false -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Addresses=true standalone.javaoptions=-Xmx1040M -XX:MaxPermSize=512M -XX:+HeapDumpOnOutOfMemoryError -ea -XX:-UseLoopPredicate

[Former Member]

Finally i am able to load the data correctly by following the below steps:

In Notepad++: Open the CSV in Notepad ++, select all the records and Convert ANSI Save It Again select all the records and clicked on "Encode It in UTF-8" option and dropped it on my server and it is loaded the data as expected.

[arvind-kumar_avinash]

Exactly, this is what I told you in the beginning that the problem is with the files which are being dropped into the Hot Folder.

[Former Member]

Files are in UTF-8 only but system is not able to recognize them.

[arvind-kumar_avinash]

I have updated my answer with the methods to check if your file is UTF-8 encoded or not. I hope it will help you find out if your files are encoded with UTF-8 or not.

[Former Member]

No when i changed the files to ANSI from UTF-8 only files are loading with out any issues along with symbols.

[arvind-kumar_avinash]

clicked on "Encode It in UTF-8" option and dropped it on my server and it is loaded the data as expected.

This is what you mentioned earlier. By the way, not 100% sure but I think ANSI should also work.