SAP IQ LOAD TABLE UTF8 ISSUES

[phil-stothard]

Struggling with a number of issues regarding multiligual data. And in particular loading through the LOAD TABLE function when the data is (in theory) UTF8. It seems to be just loading directly as ANSI, so as two characters rather than one.

There used to be an ENCODING option within LOAD TABLE but that must have been deprecated - most online comments suggest using that, but it is not in 16.1

It is probably something really simple but I'm going around in circles. The data is coming from SQL Server as output to to a text file. Looking at the hex view of the file, there are for example pairs of hex values for some european accented characters

When viewing the file in notepad++ it is fine and has the appropriate o with a line through character

But when loaded to IQ it ends up as two separate characters

I have other tables in the same DB where we are loading direct from sql server for INSERT LOCATION and having set up some environment variables relating to locale, that works fine so the IQ tables are supporting UTF8. The collation on the DB is 1252LATIN1.

I'm sure this is really simple and just me missing something obvious. Any ideas!?

[tayeb_hadjou]

Outputs of

select db_property('collation');

• In dbisql connection executed Load table :

select connection_property('Charset') as load_charset;

• In dbisql connection displaying data loaded :

select connection_property('Charset') as result_charset;

• In client dbisql client machine:
  echo $LANG
  
  Also, have you used IQ dbisql or Open client isql to load and see results?

[tayeb_hadjou]

What's the result of select db_property('collation'); ?

Try connect using client charset UTF-8 and test query to see if data are correct.

in connection parameters add
CharSet=UTF-8

See
CharSet (CS) Connection Parameter

Once connected, double check the actual connection charset
select connection_property('Charset') as client_charset;

If UTF8, then query and check data.

If you still see incorrect data, then I would advise check IQ logs covering the load test (iqmsg, srvlog, ZR_request logging).

[phil-stothard]

results viewed with iq dbisql -have loaded other data into the same table direct from SQL server table via INSERT LOCATION and they look fine (once the LANG/LCALL environment variable set to us_english.utf8)

load_charset : windows-1252

results_charset : windows-1252

LANG env variable is us_english.utf8 (system level)

Thanks