SAP CPI - Amazon S3 Integration - Groovy Header Si...

rhviana · ‎02-20-2021

Hello Folks,

One more interesting blog and sharing the knowledge and experience with you, basically let's follow the topics and instructions during the reading.

Tip in case that you still on-premise (PI or PO)

During my research about this signature I found a very good blogs from fantastic development, please take look below.

Integrating Amazon Simple Storage Service (Amazon S3) - rajeshps

File Upload in AWS S3 using REST API - asutoshmaharana2326

Agenda:

Introduction

Scenario and Integration Perspective

Amazon S3 Bucket service

Adapters available in SAP CPI
1. Open Connector Amazon
2. HTTPS
3. Amazon Web Services Adapter for SAP Integration Suite

Groovy to convert XML to JSON

AWS Signature via groovy - Method PUT
1. Problem to use MessageDigest - SHA256 - in groovy
2. Method GET

Integration Flow in SAP CPI using HTTPS Adapter
1. Adapter configuration is basic
2. Groovy - HTTP Exception
3. Possible Errors

Integration Flow in SAP CPI using AmazonWebService Adapter

SOAP UI Test

Introduction:

In below blog I would like to share how we can integrate Amazon S3 Bucket service using the HTTPS standard adapter doing the AWS Signature via groovy.

Also I will mention the others possibilities in case of no license to use Open Connector for Amazon or if you are not aware about the Amazon Web Service Adapter release 09 Feb 2021.

What I will not cover in the blog ?

The whole setup of the Amazon S3 service detail - Why ? You can check in the blog of sriprasad.shivarambhat those details, no make sense repeat.

SDK'S available from the Amazon with ready code to be used. (There is for JavaScript but not for groovy)

How to deploy the new adapter release by SAP.

Scenario and Integration perspective:

The scenario is the integration of SAP MDG sending a custom XML related with material state change (create, delete, change).

The SAP CPI will be responsible to receive the call, parse the XML to JSON using a custom groovy development and also create the signature of AWS in groovy and send the binary file using the HTTPS adapter with the PUT method to store in the Amazon S3 Bucket.

Further in this blog you will have access to the groovy codes.

*** Important information the code must be change for the correct host information details of your AWS Server ***

Amazon S3 Bucket service

An Amazon S3 bucket is a public cloud storage resource available in Amazon Web Services' (AWS) Simple Storage Service (S3), an object storage offering. Amazon S3 buckets, which are similar to file folders, store objects, which consist of data and its descriptive metadata.

Basically the S3 bucket act on-cloud repository

S3 Bucket vs traditional SFTP:

I recommend you to read the advantages to think - GO TO CLOUD in your business, not only for storage perspective, please take look in the link below with the main differences between S3 Bucket (Cloud) and the traditional on-premise SFTP server.

S3 vs SFTP

Important to mention that Amazon provides SFTP on-cloud also, we are not going to explore this topic in this blog.

Adapters available in SAP CPI

Open Connector Amazon

For this adapter there is a fantastic blog from sriprasad.shivarambhat explaining clearly how to setup and make the configuration in case that your company has the license for OpenConnector for SAP CPI

Link of the Blog: Amazon Open Connector

HTTPS

This is the adapter that I choose for this scenario, basic and traditional HTTP(S) call with methods and authentication, in this case the authentication is made by groovy you will see later the configuration.

Amazon Web Services Adapter

More details about this adapter - SAP API Hub

Groovy to convert XML to JSON

Before post the code of this conversion, it is really important to mention about the SAP press - Groovy SAP Press of engswee.yeoh and vadim.klimov

Why not use the convertor XML to JSON ?

It is up to the developer but I did the groovy code.

Input XML:

<n0:BlogAwsS3HTTPSAdapter xmlns:n0="urn:sap-com:document:sap:soap:functions:mc-style">

         <Assets>

            <Documents>

               <Type>

                  <Id>REF</Id>

                  <Name>Cad 3D</Name>

               </Type>

               <Id>794290</Id>

               <Name>Cad 3D_1189.102</Name>

               <Version>AA</Version>

               <Part>000"</Part>

               <Path>/Bola/Produtos/1189.BL102.MT.T06</Path>

               <Atributes>

                  <item>

                     <Id>asset.filename</Id>

                     <Name>asset.filename</Name>

                     <Values>

                        <Valueid/>

                        <Value>Cad 3D_1189.102.dwg</Value>

                        <Uom/>

                     </Values>

                  </item>

                  <item>

                     <Id>asset.mime-type</Id>

                     <Name>name": "asset.mime-type</Name>

                     <Values>

                        <Valueid/>

                        <Value>application/octet-stream</Value>

                        <Uom/>

                     </Values>

                  </item>

               </Atributes>

            </Documents>

         </Assets>

         <Attributes>

            <item>

               <Id>TP_MEC</Id>

               <Name>Tipo de Mecanismo utilizado</Name>

               <Group/>

               <Values>

                  <item>

                     <Valueid>LOV_TP_MEC_SOLEN</Valueid>

                     <Value>Solenóide</Value>

                     <Uom/>

                  </item>

               </Values>

            </item>

            <item>

               <Id>ESTILO_DESIGN_EVIDENCIA</Id>

               <Name>Estilo e design em evidência</Name>

               <Group/>

               <Values>

                  <item>

                     <Valueid>LOV_ESTILO_DESIGN_EVIDENCIA_OUSADO</Valueid>

                     <Value>Formas e acabamentos ousados</Value>

                     <Uom/>

                  </item>

                  <item>

                     <Valueid>LOV_ESTILO_DESIGN_EVIDENCIA_NOBRE</Valueid>

                     <Value>Cores nobres e sofisticadas</Value>

                     <Uom/>

                  </item>

               </Values>

            </item>

         </Attributes>

         <EvBrand>Ceusa</EvBrand>

         <EvName>1985.C.CT-CHUV TUBO PAR CUBO - CR</EvName>

         <EvType>children</EvType>

         <Keys>

            <item>

               <Id>COD_BARRAS</Id>

               <Name>Código de barras do produto</Name>

               <Value>7894203020995</Value>

               <Group/>

            </item>

            <item>

               <Id>COD_MATERIAL</Id>

               <Name>Código do material</Name>

               <Value>1189.BL102.MT.T06</Value>

               <Group/>

            </item>

         </Keys>

      </n0:BlogAwsS3HTTPSAdapter>

Groovy Script to convert the input XML to JSON and create the name of file that will be used in the next groovy.

/**

* Creator: Ricardo Viana

* Date: 20/02/2020

**/

import com.sap.gateway.ip.core.customdev.util.Message

import groovy.json.*

import groovy.xml.*

import java.nio.charset.StandardCharsets;



def Message processData(Message message) {

    java.io.InputStream reader = message.getBody(java.io.InputStream)

    def JsonFile = new XmlParser().parse(reader)

    def builder = new JsonBuilder()

    builder {

        'id'

        'name' JsonFile.EvName.text()

        'type' JsonFile.EvType.text()

        'brand' JsonFile.EvBrand.text()

        def arrayKeys = JsonFile.Keys.item

        'Keys' arrayKeys.collect { item->

            [

                'id': item.Id.text(),

                'name': item.Name.text(),

                'value': item.Value.text(),

            ]

        }

        assets{

            documents{

                    type{

                        'id' JsonFile.Assets.Documents.Type.Id.text()

                        'name' JsonFile.Assets.Documents.Type.Name.text()

                        }

                    'id' JsonFile.Assets.Documents.Id.text()

                    'name' JsonFile.Assets.Documents.Name.text()

                    'version' JsonFile.Assets.Documents.Version.text()

                    'part' JsonFile.Assets.Documents.Part.text() 

                    'path' JsonFile.Assets.Documents.Path.text() 

                    }

                    attributes{

                        def arrayItemAssets = JsonFile.Assets.Documents.Atributes.item

                        'item' arrayItemAssets.collect { item -> 

                            [

                                'id': item.Id.text(),

                                'name': item.Name.text(),

                                'values': item.Values.collect { item2 ->

                                    [

                                      'valueId': item2.Valueid.text(),

                                      'value':  item2.Value.text(),

                                      'uom':  item2.Uom.text(),

                                    ]

                                }

                            ]

                        }    

                    }

        }

        def arrayAttributes = JsonFile.Attributes.item

        'attributes' arrayAttributes.collect { item3 ->

            [

             'id': item3.Id.text(),

             'name': item3.Name.text(),

             'group': item3.Group.text(),

             'values': item3.Values.item.collect { item4 ->

                                        [

                                       'valueId': item4.Valueid.text(),

                                       'value':  item4.Value.text(),

                                       'uom':  item4.Uom.text(),

                                        ]

                                    }

             ]

            }

    }

    

    def jsonString = JsonOutput.prettyPrint(builder.toString())

    jsonString = unescapeUnicode(jsonString)

    message.setProperty("NomeArquivo", "JsonFileBlog_SAPCPI_HttpsAdapter.json")

    message.setBody(jsonString)

    return message

}



def unescapeUnicode(def inp){

    (inp =~ /\\u([0-9a-f]{2})([0-9a-f]{2})/).each { m ->        

        def uniAsString = new String([

                                Integer.parseInt(m[1], 16),

                                Integer.parseInt(m[2], 16)

                            ] as byte[], StandardCharsets.UTF_16)

        inp = inp.replace(m[0], uniAsString)

    }

    return inp

}

Why I'm using unescapeUnicode function ?

There is some accents in some strings - Código - "ó" without it developing the groovy is not able to encode proper so this function is used to keep the encoding proper.

If you use the XML to JSON convertor, this situation not happens.

Thanks r_herrmann for the TIP.

AWS Signature via groovy - Method PUT

I will not explain in the details whole process of the signature, you can easily check in the Amazon Header Signature V4 , but I will try high light how to make it.

To generate the signature header v4 basically it is compose of 3 steps with the details of Amazon S3 Buket:

Canonical Request

String-to-Sing

Deriver-signing-key

byte [ ] signing_key =

byte[] getSignatureKey(String key, String dateStamp, String regionName, String serviceName) throws Exception {

    byte[] kSecret = ("AWS4" + key).getBytes("UTF8");

    byte[] kDate = HmacSHA256(dateStamp, kSecret);

    byte[] kRegion = HmacSHA256(regionName, kDate);

    byte[] kService = HmacSHA256(serviceName, kRegion);

    byte[] kSigning = HmacSHA256("aws4_request", kService);

    return kSigning;

}

The details of each step is clear and you will see with more details in the groovy script:

import com.sap.gateway.ip.core.customdev.util.Message;

import javax.crypto.Mac

import javax.crypto.spec.SecretKeySpec

import java.nio.charset.Charset;

import java.nio.charset.StandardCharsets;

import java.text.SimpleDateFormat

import java.lang.Object

import java.util.List;

import java.util.TimeZone;

import org.apache.commons.codec.digest.DigestUtils;

def Message processData(Message message) {

def body = message.getBody(java.lang.String) as String

//************* Hash do Body using apache commons DigestUtils sha256Hex ************* 

def hashBody = DigestUtils.sha256Hex(body)

//************* Mapping the properties - The filename was set in previous groovyScript ************* 

def map = message.getProperties()

//************* Iniciating variables *************    

String method = "PUT";    

String host = "<yourbuket>.s3.us-east-2.amazonaws.com";    

String region = "us-east-2";    

String service = "s3";

String endpoint = "s3.us-east-2.amazonaws.com";

// Read AWS access key from security artifacts. Best practice is NOT to embed credentials in code.    

def access_key = "ID_GeneratedInAWS"

def secret_key = "Key_GeneratedInAWS"

// Create a date for headers and the credential string

def now = new Date()

def amzFormat = new SimpleDateFormat( "yyyyMMdd'T'HHmmss'Z'" )

def formattedDate = new SimpleDateFormat("EEEE, MMMM dd, yyyy, hh:mm a '('zzz')'")

def stampFormat = new SimpleDateFormat( "yyyyMMdd" )

def amzDate = amzFormat.format(now)

def date_stamp = stampFormat.format(now)

//************* Canonical Request variables ************* 

String canonical_uri = "/<yourFolder>/"+map.get('NomeArquivo');

String canonical_querystring = "";

String canonical_headers = "host:" + host + "\n"+ "x-amz-content-sha256:" + hashBody + "\n" + "x-amz-date:" + amzDate + "\n";

String signed_headers = "host;x-amz-content-sha256;x-amz-date";

String canonical_request = method + "\n" + canonical_uri + "\n" + canonical_querystring + "\n" + canonical_headers + "\n" + signed_headers + "\n" + hashBody;

//************* Sing to Sing variables ************* 

String algorithm = "AWS4-HMAC-SHA256";

String credential_scope = date_stamp + "/" + region + "/" + service + "/" + "aws4_request";

String string_to_sign = algorithm + "\n" +  amzDate + "\n" +  credential_scope + "\n" + DigestUtils.sha256Hex(canonical_request);

//************* Generating the Singning Key ************* 

byte[] signing_key = getSignatureKey(secret_key, date_stamp, region, service);

//************* Generating the HmacSHA256 - Amazon ************* 

byte[] signature = HmacSHA256(string_to_sign,signing_key);

//************* Generating the Hex of the Signature ************* 

String strHexSignature = bytesToHex(signature);

//************* Generating the authorization header signed - Amazon V4 S3 Bucket ************* 

String authorization_header = algorithm + " " + "Credential=" + access_key + "/" + credential_scope + ", " +  "SignedHeaders=" + signed_headers + ", " + "Signature=" + strHexSignature;

//************* Seting the headers of HTTP call ************* 

message.setHeader("x-amz-date",amzDate);

message.setHeader("x-amz-content-sha256", hashBody)

message.setHeader("Authorization", authorization_header);

message.setHeader("Host", "<yourBucket>.s3.us-east-2.amazonaws.com");

message.setHeader("content-type", "application/json");

//************* Setting the body to be store in Amazon ************* 

message.setBody(body)

return message

}

//************* Function bytes to Hex ************* 

String bytesToHex(byte[] bytes) {

    char[] hexArray = "0123456789ABCDEF".toCharArray();           

    char[] hexChars = new char[bytes.length * 2];

    for (int j = 0; j < bytes.length; j++) {

        int v = bytes[j] & 0xFF;

        hexChars[j * 2] = hexArray[v >>> 4];

        hexChars[j * 2 + 1] = hexArray[v & 0x0F];

    }

    return new String(hexChars).toLowerCase();

}

//************* Function HmacSHA256 ************* 

byte[] HmacSHA256(String data, byte[] key) throws Exception {

    String algorithm="HmacSHA256";

    Mac mac = Mac.getInstance(algorithm);

    mac.init(new SecretKeySpec(key, algorithm));

    return mac.doFinal(data.getBytes("UTF8"));

}

//************* Function getSignature ************* 

byte[] getSignatureKey(String key, String dateStamp, String regionName, String serviceName) throws Exception {

    byte[] kSecret = ("AWS4" + key).getBytes("UTF8");

    byte[] kDate = HmacSHA256(dateStamp, kSecret);

    byte[] kRegion = HmacSHA256(regionName, kDate);

    byte[] kService = HmacSHA256(serviceName, kRegion);

    byte[] kSigning = HmacSHA256("aws4_request", kService);

    return kSigning;

}

Problems to use the Message Digest - SHA256 - in Groovy.

Lib: import java.security.MessageDigest;

Function:

def generateHex(String data) {

    MessageDigest mac = MessageDigest.getInstance("SHA-256");

    byte[] signatureBytes = mac.digest(data.getBytes(StandardCharsets.UTF_8));

    StringBuffer hexString = new StringBuffer();

    for (int j=0; j<signatureBytes.length; j++) {

        String hex=Integer.toHexString(0xff & signatureBytes[j]);

        if(hex.length()==1) hexString.append('0');

    hexString.append(hex);

    }

    String encryptedSignature = hexString.toString();

    String encryptHash = encryptedSignature.replace("-","");

    encryptHash = encryptHash.toLowerCase();

    return encryptHash;

}

Error about Security Exception in the line 83 in yellow:

To solve the problem is soft, basically import the lib import org.apache.commons.codec.digest.DigestUtils; in your code.

Lib: import org.apache.commons.codec.digest.DigestUtils;

Replace the function to generate SHA-256 HEX using the function sha256Hex from DigestUtils

def hashBody = DigestUtils.sha256Hex(body)

Link for more details: sha256Hex-java.lang.String

Different methods to develop SHA-256:

Generate SHA-256

Method GET

For the method GET there is no body in this case, so the value of content must be fix :

e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

hashBody: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Basically change the from PUT to GET:

String method = "GET".

Integration Flow in SAP CPI - HTTPS Adapter

Basically the SAP MDG is sending the data via SOAP RM - ( For test proposal I'm using the SOAP)

Groovy - XML to JSON

Groovy - Signature Header Amazon S3 V4

Syncronous call to Flow

HTTPS Adapter to S3 Bucket Amazon

Groovy HTTP Exception

Above you already could check the groovy in details, now let's present the configuration of the HTTPS adapter and the groovy of HTTP Exception.

Adapter configuration is basic

Method - PUT

Authentication - None (Groovy Signature Header Amazon S3 V4 is doing this job)

Address - <yourAwsDetailBuket> and <yourFolder>

${property.NameArquivo} - It was set in the first groovy to be Dynamic

Groovy - HTTP Exception

import com.sap.gateway.ip.core.customdev.util.Message

import java.util.HashMap

import org.w3c.dom.Node

import groovy.xml.*





def Message processData(Message message) {

    java.io.InputStream reader = message.getBody(java.io.InputStream)

    def messageLog = messageLogFactory.getMessageLog(message)

    def xHeaders = message.getHeaders()

    def map = message.getProperties()

    String logVar = map.get("enableLog")

    String sResponseCode = xHeaders.get("CamelHttpResponseCode")

    String sHtttpError = xHeaders.get ("CamelHttpResponseText")

	def ex = map.get("CamelExceptionCaught");

    if (ex!=null) {

            if (ex.getClass().getCanonicalName().equals("org.apache.camel.component.ahc.AhcOperationFailedException")) {

                messageLog.addAttachmentAsString("HTTP Response body: ", ex.getResponseBody(), "text/plain");

                String exceptionBody = "Error in the routing details in the Iflow "

                throw new Exception(exceptionBody);

                return message;

            }

    }

    else{

       String exceptionBody = "Error in the routing details in the Iflow "

       throw new Exception(exceptionBody);

       return message;

        

    }

   return message;

}

Possible erros:

There are many possibilities of errors in this HTTPS call but mainly I will present the 3 most classics for this integration.

AccessDenied:

<?xml version="1.0" encoding="UTF-8"?>

<Error>

   <Code>AccessDenied</Code>

   <Message>Access Denied</Message>

   <RequestId>WX63PW5GYS61NRYV</RequestId>

   <HostId>+pyQoa9brW6NGw0j9rMDJiz84Nhucp58we4WPbtEVRI/yzSgXhMQE9TUro97Fr5wPYDDw2QP5RQ=</HostId>

</Error>

Means that the credential user detail in AWS is without proper AWS Roles to Read and Write in the S3 bucket, check the link below:

Bucket Policy AWS

SignatureDoesNotMatch

<?xml version="1.0" encoding="UTF-8"?>

<Error>

   <Code>SignatureDoesNotMatch</Code>

   <Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>

   <AWSAccessKeyId>AKIAWXC7MCXWRGBV5QVD</AWSAccessKeyId>

</Error>

In case that you decide use the groovy script to generate the signature calculation, review all detail information that you provide.

MethodNotAllowed

You are using the HTTP method not valid, for example "POST"

<?xml version="1.0" encoding="UTF-8"?>

<Error>

    <Code>MethodNotAllowed</Code>

    <Message>The specified method is not allowed against this resource.</Message>

    <Method>POST</Method>

    <ResourceType>OBJECT</ResourceType>

    <RequestId>NEAJEF9PATJ8ADS4</RequestId>

    <HostId>Zb9ljNLrZtApEWu+Sr6fkKxFP7zy3jU3ILpr+EcQHKZfzP9nxyOdAeVr6mywTRVCls/zjMm62CA=</HostId>

</Error>

Integration Flow in SAP CPI using AmazonWebService Adapter

Basically the SAP MDG is sending the data via SOAP RM - ( For test proposal I'm using the SOAP)

Groovy - XML to JSON

Syncronous call to Flow

Amazon Adapter

Groovy Exception

Configuration of Amazon Adapter:

You must proceed with the deploy of this adapter to be visible on the list of adapters available in your SAP CPI.

Select the protocol AWS, in this case S3:

Select the region and details of your bucket and alias of credencials

Go to security materials and create alias for Access Key and Secret Key:

I will present only for access key but must make for secret key also with the same proceedure.

Processing details with informations of Operation, Directory, File Name and Content-Type

The adapter will be responsible to generate the signature header Amazon S3 V4 for you.

Let's rock now, off course using the HTTPS adapter.

First lets check what is the state of the S3 bucket:

Now I'm going to send data from SOAP UI and Trace On in the Flow:

Let me check the name of file in the trace:

Let's check the S3 Bucket again - BOOM:

To don't extend much the blog let's use the POSTMAN using the method GET to check the content if it is the same of the trace in SAP CPI.

I hope that you enjoy the read and what I presented in this blogs.

Kind Regards,

Viana.