Introduction
Not sure how many folks have reviewed the semi-recent announcement regarding Trading Partner Management in SAP Integration Suite Announcement: SAP Trading Partner Management and B2B Monitoring brand new capabilities of SAP Integr... and were as disappointed as me with the initial product delivery. I have two core issues with the approach:
- Still no support for bundling IDocs despite its longtime existence on the back-end and it's noted in SAP Help EDI Converter - that's ok... I'll tackle it on my own.
- Bloated maintenance of mapping artifacts per partner - too many folks have missed the point on EDI communication and the proposed solution is no different. Last I checked I'm still bound by the amount of time available in a day, and my company has at least a couple dozen EDI partners.
Stay tuned for a part II that will address incoming communication for 997s and have the groundwork for other transaction sets.
Now available (2022-05-22) - SAP EDI/IDoc Communication in CPI Using Bundling and Fewer Mapping Artifacts Part II
* Revision 1.1 – Adjustment to handle dynamic terminator and separator parameters in the XML to EDI converter using a combination of externalized parameters and groovy script that reads data from the PD – 2022-07-26
* Revision 1.2 – add additional error check for agreement that has not been maintained – 2023-01-13
* Revision 1.3 – add feature flag option for archiving and 997 data store operations – 2023-01-16
* Revision 1.4 – replace SFTP support for FTP support based on our partner needs (encrypted only) – 2023-01-25
* Revision 1.5 – add SFTP support again with completed script logic for VAN communication – 2023-02-02
* Revision 1.6 – remove email alerting in favor of Alert Notification Service (kudos dc01fe2c84194f24b77a64b3dbe421fd) – 2023-02-17
* Revision 1.7 – add XML validation step to ensure outgoing message meets syntax and semantic requirements – 2023-03-01
* Revision 1.8 – add Github link for package download – 2023-10-28
Prerequisites for Bulk and Determination of EDI Standard
The message function code is mapped to the appropriate field in the group segment while the package size handles the bundling, and in the other image the values are used to determine the appropriate X12 target without having to read from the TPM agreements.
Transformation Flow
The initial transformation flow is based off this older ICA template with some modifications to handle the splitting for bulk, and saving the IDoc numbers in a datastore for retrieval on incoming 997s.
Extract Basic Information and Lookup Partner Directory Information
The first two steps gather some basic IDoc information, establish the sender partner id, and generate a interchange control number, along with gathering conversion specific information and agreement data. The Groovy script taps into the partner directory to read the header information for the necessary XSLT/XSD artifacts, but with one important distinction - the central mapping artifacts are tied to our partner directory entry. The other two less important distinctions are reading binary data in json format to get extra parameters about handling - e.g. archiving, target IDoc for incoming messages, etc. and reading agreement information regarding whether an acknowledgement is required (important for following datastore step) or any extended post processing should take place. Feature flags have also been enabled for both the archiving and 997 datastore operations so they may activated/deactivated in test environments by adjusting the necessary partner directory parameters when a change is warranted. If an appropriate agreement has not been maintained, then the system will throw an exception that will result and affected IDocs being placed in '02' error status. I have included a basic sample for both the message specific parameters and the partner specific agreement information.
import com.sap.gateway.ip.core.customdev.util.Message;
import java.util.HashMap;
import com.sap.it.api.pd.PartnerDirectoryService;
import com.sap.it.api.ITApiFactory;
import groovy.json.*;
def Message processData(Message message) {
def service = ITApiFactory.getApi(PartnerDirectoryService.class, null)
if (service == null){
throw new IllegalStateException("Partner Directory Service not found")
}
// Read partner data, and EDI message information for conversion and
// communication purposes
def headers = message.getHeaders()
def SenderPid = headers.get("SenderPid")
def rcvprn = headers.get("SAP_IDoc_RCVPRN")
def ReceiverPid = service.getPartnerId("SAP SE", "IDOC", rcvprn)
message.setHeader("ReceiverPid", ReceiverPid)
def mestyp = headers.get("SAP_IDoc_MESTYP")
def idoctyp = headers.get("SAP_IDoc_IDOCTYP")
def std = headers.get("SAP_IDoc_STD")
def stdmes = headers.get("SAP_IDoc_STDMES")
def stdvrs = headers.get("SAP_IDoc_STDVRS")
def test = headers.get("SAP_IDoc_TEST")
// Retrieve our qualifier and id for ISA envelope and pass usage indicator,
// and also read receiving partner information
def our_x12qual = service.getParameter("x12_qualifier", SenderPid, java.lang.String.class)
message.setProperty("OurX12Qualifier", our_x12qual)
def our_x12id = service.getParameter("x12_id", SenderPid, java.lang.String.class)
message.setProperty("OurX12Id", our_x12id)
message.setProperty("UsageIndicator", test == "X" ? "T" : "P")
def x12qual = service.getParameter("x12_qualifier", ReceiverPid, java.lang.String.class)
message.setProperty("X12Qualifier", x12qual)
def x12id = service.getParameter("x12_id", ReceiverPid, java.lang.String.class)
message.setProperty("X12Id", x12id)
def x12SegmentTerminator = service.getParameter("x12_segment_terminator", ReceiverPid, java.lang.String.class)
message.setProperty("X12SegmentTerminator", x12SegmentTerminator)
def x12CompositeSeparator = service.getParameter("x12_composite_separator", ReceiverPid, java.lang.String.class)
message.setProperty("X12CompositeSeparator", x12CompositeSeparator)
def x12ElementSeparator = service.getParameter("x12_element_separator", ReceiverPid, java.lang.String.class)
message.setProperty("X12ElementSeparator", x12ElementSeparator)
def x12RepetitionSeparator = service.getParameter("x12_repetition_separator", ReceiverPid, java.lang.String.class)
message.setProperty("X12RepetitionSeparator", x12RepetitionSeparator)
// Read extra parameters for message handling - e.g. archiving, message target, etc.
def slurper = new JsonSlurper()
def isArchivingActive = service.getParameter("isArchivingActive", SenderPid, java.lang.String.class)
if(isArchivingActive == "true") {
def msgParamsBinary = service.getParameter(stdmes, SenderPid, com.sap.it.api.pd.BinaryData)
if(msgParamsBinary != null) {
def msgParameters = slurper.parse(msgParamsBinary.getData(), "UTF-8")
if(msgParameters.Outbound?.ArchiveMessage == "true") {
message.setHeader("ArchiveMessage", msgParameters.Outbound.ArchiveMessage)
def folder = headers.get("ArchiveFolder")
message.setHeader("ArchiveFolder", folder + ReceiverPid + "/")
} else {
message.setHeader("ArchiveFolder", null)
}
}
}
// Setup header variables from PD for conversion handling (AT items)
def msgInfo = "ASC-X12_" + stdmes + "_" + stdvrs
def preproc, mapping, postproc, rec_conversion
preproc = mestyp + "." + idoctyp + "_preproc"
if(std == "X") {
mapping = mestyp + "." + idoctyp + "_to_" + msgInfo
postproc = msgInfo + "_postproc"
rec_conversion = msgInfo
}
message.setHeader("PREPROC_XSLT", "pd:" + SenderPid + ":" + preproc + ":Binary")
message.setHeader("MAPPING_XSLT", "pd:" + SenderPid + ":" + mapping + ":Binary")
message.setHeader("POSTPROC_XSLT", "pd:" + SenderPid + ":" + postproc + ":Binary")
message.setHeader("REC_CONVERSION_XSD", "pd:" + SenderPid + ":" + rec_conversion + ":Binary")
// Retrieve partner specific information regarding agreement information for
// extended post processing and acknowledgements
def agreementParamsBinary = service.getParameter("Agreements", ReceiverPid, com.sap.it.api.pd.BinaryData)
if(agreementParamsBinary != null) {
def agreementParameters = slurper.parse(agreementParamsBinary.getData(), "UTF-8")
def msgAgreement = agreementParameters.Agreements.Outbound.find{ it.Message == msgInfo }
if(msgAgreement == null) {
throw new IllegalStateException("Partner agreement not maintained")
}
// 997 datastore can be disabled per landscape like feature flag and 997 data will not be stored
// if datastore is not marked as active
def is997DatastoreActive = service.getParameter("is997DatastoreActive", SenderPid, java.lang.String.class)
message.setProperty("AcknowledgementRequired", is997DatastoreActive == "true" ? msgAgreement?.AcknowledgementRequired : "false")
message.setProperty("AcknowledgementRequested", msgAgreement?.AcknowledgementRequired == "true" ? "1" : "0")
if(msgAgreement?.DoExtendedPostProcessing == "true") {
message.setProperty("DoExtPostprocessing", msgAgreement.DoExtendedPostProcessing)
message.setHeader("EXT_POSTPROC_XSLT", "pd:" + ReceiverPid + ":ext_" + msgInfo + "_postproc:Binary")
}
} else {
throw new IllegalStateException("Partner agreement not maintained")
}
return message
}{
"Outbound": {
"ArchiveMessage": "true"
},
"Inbound": {
"Target": "INVOIC.INVOIC02"
}
}{
"Agreements": {
"Inbound": [],
"Outbound": [
{
"Message": "ASC-X12_856_004010",
"AcknowledgementRequired": "true",
"DoExtendedPostProcessing": "false"
}
]
}
}Write to Datastore (If applicable)
Responsible for saving the IDoc #s into the datastore with the interchange control number as the entity id in the event that an acknowledgement is both expected and required (Necessary for STATUS.SYSTAT01 communication later). The following shows the established configuration for the write step to the datastore, and the XSL mapping step has not been included because it is your basic run of the mill XSL transform.
Determine Envelope Type
The retrieval of the partner directory information is followed by the interchange mapping which has been configured as a local integration process with a general splitter and gather step sandwiching the necessary mapping steps. It also includes one additional XSLT step not found in the older standard ICA content that manages the segment counter in the S_SE segment, and was referenced from Integration Advisor: Functions at target side – Ordinal number for line items and segment count (Mapping was enhanced to include inserting values for ST/SE counters for XML validation - true values are inserted later in Transaction Count mapping step shown below. The other option is to hardcode valid values in the mappings in the integration advisor). Once the interchange mapping is complete the enveloped is constructed using the same mechanism as the older ICA template, but with the inclusion of a router based on version number. The route for 004010 is the default with the expression for the other route as follows:
Execute XML to EDI Converter
The EDI converter parameters for the segment terminator, and each of the qualified separators will be recorded and read from the partner directory. Each individual parameter is externalized in the XML to EDI converter step, and configured as a property that corresponds to a partner directory read in the Lookup PD script.
Transactional Control Numbers and Total Transaction Count
An extra XSL transform that is a variation of the segment counter XSL with some additional logic to output transactional control numbers based on position + 1, and also pad ISA ids with trailing spaces.
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="3.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="xml" version="1.0" encoding="UTF-8" indent="yes"/>
<xsl:template match="/">
<xsl:apply-templates select="*"/>
</xsl:template>
<!-- ============================================================================================= -->
<!-- Template: Enter the count of segments -->
<!-- ============================================================================================= -->
<xsl:template match="D_97">
<D_97>
<xsl:value-of select="count(//*[name() = 'S_ST'])"/>
</D_97>
</xsl:template>
<!-- ============================================================================================= -->
<!-- Template: Pad IDs for both sender and receiver in ISA envelope-->
<!-- ============================================================================================= -->
<xsl:template match="D_I06">
<xsl:variable name="Sid" select="."/>
<D_I06><xsl:call-template name="padStr">
<xsl:with-param name="str" select="$Sid"/>
<xsl:with-param name="chr" select="' '"/>
<xsl:with-param name="len" select="15"/>
</xsl:call-template></D_I06>
</xsl:template>
<xsl:template match="D_I07">
<xsl:variable name="Rid" select="."/>
<D_I07><xsl:call-template name="padStr">
<xsl:with-param name="str" select="$Rid"/>
<xsl:with-param name="chr" select="' '"/>
<xsl:with-param name="len" select="15"/>
</xsl:call-template></D_I07>
</xsl:template>
<!-- ============================================================================================= -->
<!-- Template: Determine Transaction Set Control Number -->
<!-- ============================================================================================= -->
<xsl:template match="D_329">
<D_329><xsl:value-of select="format-number(count(../../preceding-sibling::*)+1, '000000000')"/></D_329>
</xsl:template>
<!-- ============================================================================================= -->
<!-- Template: Consume and produce remaining nodes -->
<!-- ============================================================================================= -->
<xsl:template match="node() | @*">
<xsl:copy>
<xsl:apply-templates select="node() | @*"/>
</xsl:copy>
</xsl:template>
<!-- ============================================================================================= -->
<!-- Template: String Padding on Right -->
<!-- ============================================================================================= -->
<xsl:template name="padStr">
<xsl:param name="str"/>
<xsl:param name="chr"/>
<xsl:param name="len"/>
<xsl:variable name="pad">
<xsl:for-each select="1 to $len">
<xsl:value-of select="$chr" />
</xsl:for-each>
</xsl:variable>
<xsl:value-of select="substring(concat($str,$pad),1,$len)"/>
</xsl:template>
</xsl:stylesheet>The Last Step
The final content modifier is there to remove some header information from the envelope step where the system complains about improper CRLF characters and set a header for pid which is used in the communication flow.
Communication Flow
The communication flow is setup with a JMS queue with simple retry every 15 minutes settings and an exception sub-process to generate an error that is tracked in the Alert Notification Service - see here for ANS setup https://blogs.sap.com/2022/02/01/connect-sap-platform-integration-with-alert-notification-service-cl.... For the most part this flow is WYSIWYG. Maybe you would ask... why the partner directory lookup again, but that is simply for separating transformation logic from communication logic which will be obvious in the script code. The FTP communication only supports implicit or explicit encryption via the dynamic setting in the FTP receiver adapter, and the IDoc work will be referenced in part II.
Partner Directory Script for Communication
Gather adapter information and, if necessary, details related to third party communication for AS2 and FTP communication. The final piece creates a header for the archive file name to be passed to the FTP adapter for processing.
import com.sap.gateway.ip.core.customdev.util.Message;
import java.util.HashMap;
import com.sap.it.api.pd.PartnerDirectoryService;
import com.sap.it.api.ITApiFactory;
def Message processData(Message message) {
def service = ITApiFactory.getApi(PartnerDirectoryService.class, null)
if (service == null){
throw new IllegalStateException("Partner Directory Service not found")
}
// Merge Partner Directory data into message header
def headers = message.getHeaders()
def pid = headers.get("pid")
def stdmes = headers.get("SAP_EDI_Message_Type")
def intchg = headers.get("SAP_EDI_Interchange_Control_Number")
def adapter = service.getParameter("AdapterType", pid, java.lang.String.class)
message.setProperty("adapter", adapter)
if(adapter == "AS2") {
def url = service.getParameter("ReceiverUrl", pid, java.lang.String.class)
message.setProperty("Url", url)
def recipientAS2 = service.getParameter("AS2_id", pid, java.lang.String.class)
message.setProperty("RecipientAS2", recipientAS2)
def publicKey = service.getParameter("PublicKeyAlias", pid, java.lang.String.class)
message.setProperty("PublicKeyAlias", publicKey)
def compress = service.getParameter("SAP_AS2_Outbound_Compress_Message", pid, java.lang.String.class)
message.setHeader("SAP_AS2_Outbound_Compress_Message", compress)
def signMessage = service.getParameter("SAP_AS2_Outbound_Sign_Message", pid, java.lang.String.class)
message.setHeader("SAP_AS2_Outbound_Sign_Message", signMessage)
def signAlgorithm = service.getParameter("SAP_AS2_Outbound_Signing_Algorithm", pid, java.lang.String.class)
message.setHeader("SAP_AS2_Outbound_Signing_Algorithm", signAlgorithm)
def encryptMessage = service.getParameter("SAP_AS2_Outbound_Encrypt_Message", pid, java.lang.String.class)
message.setHeader("SAP_AS2_Outbound_Encrypt_Message", encryptMessage)
def encryptAlgorithm = service.getParameter("SAP_AS2_Outbound_Encryption_Algorithm", pid, java.lang.String.class)
message.setHeader("SAP_AS2_Outbound_Encryption_Algorithm", encryptAlgorithm)
def mdnType = service.getParameter("SAP_AS2_Outbound_Mdn_Type", pid, java.lang.String.class)
message.setHeader("SAP_AS2_Outbound_Mdn_Type", mdnType)
def mdnRequestSign = service.getParameter("SAP_AS2_Outbound_Mdn_Request_Signing", pid, java.lang.String.class)
message.setHeader("SAP_AS2_Outbound_Mdn_Request_Signing", mdnRequestSign)
def mdnSignAlgorithm = service.getParameter("SAP_AS2_Outbound_Mdn_Signing_Algorithm", pid, java.lang.String.class)
message.setHeader("SAP_AS2_Outbound_Mdn_Signing_Algorithm", mdnSignAlgorithm)
def mdnVerifySign = service.getParameter("SAP_AS2_Outbound_Mdn_Verify_Signature", pid, java.lang.String.class)
message.setHeader("SAP_AS2_Outbound_Mdn_Verify_Signature", mdnVerifySign)
def mdnRequestMic = service.getParameter("SAP_AS2_Outbound_Mdn_Request_Mic", pid, java.lang.String.class)
message.setHeader("SAP_AS2_Outbound_Mdn_Request_Mic", mdnRequestMic)
def mdnVerifyMic = service.getParameter("SAP_AS2_Outbound_Mdn_Verify_Mic", pid, java.lang.String.class)
message.setHeader("SAP_AS2_Outbound_Mdn_Verify_Mic", mdnVerifyMic)
} else if(adapter == "SFTP" || adapter == "FTP") {
def host = service.getParameter("host", pid, java.lang.String.class)
message.setHeader("host", host)
def directory = service.getParameter("directory", pid, java.lang.String.class)
message.setHeader("directory", directory)
def credentials = service.getParameter("credentials", pid, java.lang.String.class)
message.setHeader("credentials", credentials)
message.setHeader("CamelFileNameOnly", stdmes + "_" + intchg + ".edi")
if(adapter == "FTP") {
def encryption = service.getParameter("SAP_FtpEncryption", pid, java.lang.String.class)
if(encryption == "ftps" || encryption == "ftpes") {
message.setProperty("SAP_FtpEncryption", encryption)
} else {
message.setHeader("adapter", "")
}
}
}
// Set archive file name if archiving is activated for message
def archive = headers.get("ArchiveMessage")
if(archive == "true") {
message.setHeader("ArchiveFile", stdmes + "_" + intchg + ".edi")
}
return message
}Archive Handling
The default route is for no archiving, but in the event archiving is required, then sequential multicast is used with the first branch being the archive step. This approach ensures that the archive is always saved first (internal FTP with overwrite) and you never get orphaned exchanges where no archive has been saved. I believe it is generally required in most, if not all cases, to save several years in the case of 810s for audit purposes.
Conclusion
Not a whole lot to say here, but rather to show it can be done. The important note being that it is not supported so take it under advisement and do so at your own risk. I encourage folks to comment, like, and ask any questions (Some detail was glossed over for brevity). Cheers!
References
https://blogs.sap.com/2021/12/17/announcement-sap-trading-partner-management-and-b2b-monitoring-bran...
https://help.sap.com/docs/CLOUD_INTEGRATION/987273656c2f47d2aca4e0bfce26c594/707973f9bfb1419eb806287...
https://api.sap.com/integrationflow/IDOCToX12_4010_Outbound
https://blogs.sap.com/2020/12/10/integration-advisor-functions-at-target-payload-ordinal-number-for-...
https://blogs.sap.com/2016/08/11/hcihcp-isidoc-adapter-deciphered-part-4-trigger-idoc-from-hci-to-sa...
https://blogs.sap.com/2022/02/01/connect-sap-platform-integration-with-alert-notification-service-cl...
https://github.com/ryanc35/X12EDIinCPI
- SAP Managed Tags:
