UTF8 encoding Issue in the conversion program

Hi all,

Our requirement is to get data from a file stored in application server and update the SAP database.

The flat file has some special characters such as u2122, ® etc.

In the conversion program, we are using the encoding format as UTF-8. This means it will only handle UTF-8 file ( byte order mark (BOM) 8).

After running the conversion program, we came to know that the special character u2122 is getting replaced wtih #.

And when we modified the format of flat file to UTF-8 instead of ANSI or UTF-16 or 32.

We were able to get the trademark symbol in correct look in addition with other special characters.

Since this involves a manual of processing modifying the flat file to UTF-8 format and then running the conversion program for getting all the special characters in good look, we are thinking to make this process "automatic" in the conversion program itself i.e. whatever the format of flat file may be, we would convert that file to UTF with BOM 8 format and then go ahead with the next updating processes.

So, can anybody let me know if we can make this an automatic process? Can we convert each non UTF-8 record read from flat file to UTF-8 record in the program itself by any means and then update that record in the database? Is there any FM for this? Or can we convert the flat file to UTF-8 file programmatically irrespective of the original file formats?

Any inputs on this would be greatly appreciated.

Thanks,

Shamim