How to decode UTF-16LE in SAP CPI

[former_member751867]

Hi All,

we have a requirement to Fetch encoded csv file from sender SFTP Server and need to decode characters UTF-16LE in to UTF-8.

I have tried below Groovy script.

Input:

import com.sap.gateway.ip.core.customdev.util.Message

import java.nio.charset.StandardCharsets

import java.nio.charset.Charset

def Message processData(Message message) {

def body = message.getBody(java.lang.String);

message.setBody(body.getBytes("UTF-8"));

return message

}

output:

1 0 2 4 5 7 9 , B e l g i u m , c o r p o r a t e

1 6 4 2 8 1 3 , E u r o p e, c o r p o r a t e

3 5 2 6 8 3 1 , I t a l y , i n d u s t r y

https://answers.sap.com/questions/13416178/how-to-decode-utf-16le-in-sap-cpi.html

https://answers.sap.com/questions/13416178/how-to-decode-utf-16le-in-sap-cpi.html

output of the data is coming with unwanted spaces and symbols.

could anyone please suggest with right solution.

Regards

Ashok

[Ivan-Mirisola]

Hi ashok_kumar01,

As Mike already suggested, you should always enforme UTF-8 content in yout iFlows. Therefore, ask the file owners to encode that file in UTF-8 format instead of UTF-16LE - so you can avoid such conversions.

If that is absolutely not possible, please try the following code. I didn't make it to work on CPI - just with a local groovy development - do adapt it to your needs.

import java.nio.ByteBuffer
import java.nio.CharBuffer
import java.nio.charset.CharacterCodingException
import java.nio.charset.CharsetDecoder
import java.nio.charset.CharsetEncoder
import java.nio.charset.CodingErrorAction
import java.nio.charset.StandardCharsets

import org.apache.groovy.parser.antlr4.GroovyParser.CatchClauseContext

class MyGroovyClass {
	static void main(def args){
		byte[] bStrUtf8 = "Eu não sou o José".getBytes("UTF-8");
		byte[] bStrUtf16le = "Eu não sou o José".getBytes("UTF-16LE");

		System.out.println("Converted String UTF-8: " + bStrUtf8.toString());
		System.out.println("Converted String UTF-16LE: " + bStrUtf16le.toString());
		
		System.out.println("Reencoding string into UTF-8: " + getUTF8StringFrom16LE(bStrUtf8)); // This should not work as array is encoded in UTF8 already
		System.out.println("Reencoding string into UTF-8: " + getUTF8StringFrom16LE(bStrUtf16le));

		System.out.println("Reencoding string into UTF-16LE: " + getUTF16LEStringFrom8(bStrUtf8)); 
		System.out.println("Reencoding string into UTF-16LE: " + getUTF16LEStringFrom8(bStrUtf16le)); // This should not work as array is encoded in UTF16LE already
	}
	
	public static String getUTF8StringFrom16LE(byte[] bytes) {
		ByteBuffer buffer = ByteBuffer.wrap(bytes);
		CharsetDecoder decoder = StandardCharsets.UTF_16LE.newDecoder();
		decoder.onMalformedInput(CodingErrorAction.REPORT);
		decoder.onUnmappableCharacter(CodingErrorAction.REPORT);
		
		CharsetEncoder encoder = StandardCharsets.UTF_8.newEncoder();
		encoder.onMalformedInput(CodingErrorAction.REPORT);
		encoder.onUnmappableCharacter(CodingErrorAction.REPORT);

		try {
			char[] b16 = decoder.decode(buffer).getChars();
			CharBuffer buffer8 = CharBuffer.wrap(b16);
			
			byte[] s8 = encoder.encode(buffer8).array();
			return s8.toString();

		} catch (CharacterCodingException e) { // Error in UTF-8
			return new String("Decoder Error: " + e);
		} catch (Exception e1) {
			return new String("Error: " + e1);
		}
	}
	
	
	public static String getUTF16LEStringFrom8(byte[] bytes) {
		ByteBuffer buffer = ByteBuffer.wrap(bytes);
		CharsetDecoder decoder = StandardCharsets.UTF_8.newDecoder();
		decoder.onMalformedInput(CodingErrorAction.REPORT);
		decoder.onUnmappableCharacter(CodingErrorAction.REPORT);
		
		CharsetEncoder encoder = StandardCharsets.UTF_16LE.newEncoder();
		encoder.onMalformedInput(CodingErrorAction.REPORT);
		encoder.onUnmappableCharacter(CodingErrorAction.REPORT);

		try {
			char[] b8 = decoder.decode(buffer).getChars();
			CharBuffer buffer16le = CharBuffer.wrap(b8);
			
			byte[] s16 = encoder.encode(buffer16le).array();
			return s16.toString();

		} catch (CharacterCodingException e) { // Error in UTF-16LE
			return new String("Decoder Error: " + e);
		} catch (Exception e1) {
			return new String("Error: " + e1);
		}
	}
}
<br>

If you run the code above, you will get the following results:

Converted String UTF-8: [69, 117, 32, 110, -61, -93, 111, 32, 115, 111, 117, 32, 111, 32, 74, 111, 115, -61, -87]
Converted String UTF-16: [69, 0, 117, 0, 32, 0, 110, 0, -29, 0, 111, 0, 32, 0, 115, 0, 111, 0, 117, 0, 32, 0, 111, 0, 32, 0, 74, 0, 111, 0, 115, 0, -23, 0]
Reencoding string into UTF-8: Decoder Error: java.nio.charset.MalformedInputException: Input length = 1
Reencoding string into UTF-8: [69, 117, 32, 110, -61, -93, 111, 32, 115, 111, 117, 32, 111, 32, 74, 111, 115, -61, -87, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Reencoding string into UTF-16LE: [69, 0, 117, 0, 32, 0, 110, 0, -29, 0, 111, 0, 32, 0, 115, 0, 111, 0, 117, 0, 32, 0, 111, 0, 32, 0, 74, 0, 111, 0, 115, 0, -23, 0]
Reencoding string into UTF-16LE: Decoder Error: java.nio.charset.MalformedInputException: Input length = 1
<br>

Please notice that I am printing the char values, not the actual string just for comparisons.

Hope it helps you.

Best regards,
Ivan

[ZhangMingquan]

Could you please try to remove the spaces after each character?

You can also refer to SAP Help Avoiding Encoding Issues