Data Hub Special Characters

[Former Member]

Hi Experts,

we figured out that the data hub seems to have a problem with special characters (i.e. ä,ö,ü). Within SAP Integration we have classification attributes that consists of these characters. The canonical items do have '?' instead of the special characters. With debugging we found out that the special characters arrive correctly within the datahub idoc input channel.

Does the data hub fully support UTF-8 characters? Is this a bug?

Kind regards, Stephan

[Former Member]

Finally we found the problem. The data hub internally generates XML files and saves them as BLOB data within the database. Therefore the start parameter -Dfile.encoding=UTF-8 has to be set for some environments. Otherwise the issue above rises. Bad mistake, initially nobody would have thought about file encoding when writing to the database.

Thanks for the hints!

[rafal_gdowski]

Good find. You should accept your answer for the future users that have the same issue.

[Former Member]

Did you put the -Dfile.encoding=UTF-8 property to the start parameters of Tomcat server and MySQL server? I did put that in start parameters for both servers but I am still getting garbage characters when the value is something like '®'. It stores '®' in the database.

[Former Member]

Hi,

we could only fix this issue by embedding the data hub into the hybris installation. For production use this is not recommended so that we are searching for another solution.

We found out that the issue seems to be somehow dependant on the operating system as the same tomcat with the same configuration and the same data hub is working with the same database on a windows system. Also the executing user is the same for the embedded version. The only difference is the tomcat running on another OS.

Hope this helps other people with that issue. Maybe we are the only one so far with that constellation of system components (SuSE, Tomcat 7, Oracle 11.2).

Kind regards, Stephan

[former_member224482]

Here is a similar question with screen shots : https://experts.hybris.com/questions/4985/unicode-character-processing.html

[Former Member]

I would suggest you double check your collation settings on your databases - both the one hybris uses and the one Data Hub uses. It might just be set incorrectly. If all parties handle characters as UTF-8 it works perfectly. Otherwise you can check the entries in the Data Hub database and / or debug the REST calls. If you have enough logging enabled you could check the URL that hybris calls after Data Hub sent it the notification that data to be imported is available.

[rafal_gdowski]

The Data Hub fully supports UTF-8 characters. Can you verify that your special characters are stored in the database properly (in the RawItem table) after they are loaded through the input channel? We have tests that validate special characters going all the way to the hybris platform, so it could be that something else is affecting your data during processing.