Introducing the new pipeline concept in Cloud Inte...

alex_bundschuh · ‎03-19-2024

We have lately published the so-called Pipeline Concept for Cloud Integration as part of the Migration Guide for SAP Process Orchestration. This can be seen as another building block supporting you in your transition to the cloud. The pipeline concept has been a joint effort of development, consulting, Center of Excellence experts, product management, and selected customers with whom we have validated the concept at a very early stage.

In the Pipeline Concept documentation, we describe the concept in detail. Furthermore, you find an integration package with the set of integration flows and a groovy script collection on the SAP Business Accelerator Hub to setup your pipeline on Cloud Integration.

Feel free to apply the concept to your migration projects. You can use the material that we provide as template, feel free to adapt to your needs, hence we call it a concept and not a final solution. If you are a partner supporting your customers within their migration projects or if you even have your own migration tool in your portfolio, feel free to apply the concept for your own tool.

What are the benefits?

Cloud integration is very flexible in terms of modeling and running your integration scenarios allowing the design of a rich variety of integration patterns. SAP Process Orchestration in contrary has a very strict pipeline designed for reliable message processing providing decoupling, splitting and restarting capabilities out of the box guaranteeing Exactly Once processing. Those requirements are still valid for interfaces in Cloud Integration and are all addressed with the pipeline concept.

In a nutshell, key achievements of the pipeline concept are as follows:

Ensures decoupling and Exactly Once processing for asynchronous processing, also for split scenarios and in case of restart
Provides sophisticated restart capabilities, i.e., automatic retries as well as manual retry options
Simplifies operations by separating errors into different generic queues separated by routing, mapping and receiver system delivery
Requires lower number of JMS queues to take into account the resource limits of an SAP Integration Suite tenant which at the end also simplifies operations
Allows isolation and tuning of parallelization for individual receiver systems
Allows reusability of artifacts across multiple flows by using generic flow concept
Simplifies integration flow model especially for content based router and recipient list scenarios
Simplifies configuration on the backend sender side, e.g., just one ALE port on SAP S/4HANA system required
Supports packaging in IDoc and proxy, multi-message mappings, and maintain order at runtime in a generic manner

What is it all about?

The pipeline concept allows you to set up your asynchronous integration scenarios in Cloud Integration in a similar way how messages are processed in SAP Process Orchestration, namely in pipelines. Other than in Cloud Integration where you are very flexible in orchestrating the message flows, pipelines in SAP Process Orchestration are rather static, i.e., each pipeline consists of a fixed number of pipeline steps whereas the order is fixed, e.g., for a message processed within the SAP Process Orchestration runtime, first the receiver is determined, then the interface, and then the mapping is carried out.

The pipeline concept is mainly addressed to existing SAP Process Orchestration customers which plan to migrate their scenarios to the Cloud Integration capability of SAP Integration Suite. However, it can also be generally applied to any asynchronous integration scenario that you like to implement and run on Cloud Integration. In case of the migration use case, it can be seen as an alternative approach or in conjunction with the individual migration of single integration scenarios supported by the migration tool within SAP Integration Suite or migration tools provided by partners to overcome its shortcomings.

The pipeline concept may not apply to all customers, it depends on your individual situation, i.e., the number or the type of integration scenarios that you have. If you have a low number of integration scenarios to be migrated to Cloud Integration, then you may rather opt for creating individual integration flows for each of your source integration scenarios. If you have however a very large number of scenarios, the pipeline concept may help you to accelerate your migration project itself and to reduce effort and cost in terms of operating and monitoring your integration scenarios in the Cloud Integration.

One of the shortcomings that we address with the pipeline concept is the JMS queue handling in Cloud Integration for asynchronous integration scenarios. Resources on an SAP Integration Suite tenant are limited, so you need to properly model your integration flows in such a way to best suit the resource limits of your tenants, see Run an Integration Flow Under Well-Defined Boundary Conditions for instance. If you migrate your asynchronous integrated configuration scenarios in SAP Process Orchestration to integration flows in Cloud Integration one-by-one and use an own JMS queue for each integration flow to ensure exactly once delivery, you end up in a very high number of JMS queues. The pipeline concept reduces the number of JMS queues required to run your integration scenarios on Cloud Integration. A proper usage of JMS queues and ProcessDirect adapters and relying on the Partner Directory to be able to dynamically configure your integration flows, help to reduce the number of required JMS queues to four. If you count in four more JMS queues for the retry and dead letter handling, you end up at eight JMS queues required.

What’s the approach?

The pipeline concept defines a sequence of integration flows each representing a pipeline step. We distinguish between a set of generic integration flows which can be commonly used across all scenarios and scenario-specific integration flows. The generic integration flows are used across all integration scenarios and hence need to be deployed only once. The scenario-specific integration flows handle the scenario-specific message conversions and mappings. They can be created as copy of the templates provided.

In general, the integration flows are decoupled using JMS queues. This ensures that messages can be restarted which is a prerequisite for guaranteed delivery. To call the scenario-specific integration flows however we use the ProcessDirect adapter. Replacing the ProcessDirect connection with a JMS connection was not an option because then we would end up again in far too many JMS queues.

One key element of the pipeline concept is the usage of the Partner Directory. The Partner Directory on the one hand helps to define the message processing behavior, e.g., you can define scenario-specific maximum number of retries or receiver-specific outbound queues. On the other side, you use the Partner Directory to dynamically configure the generic integration flows. Latter helps to reduce the number of required JMS queues.

Another key element is the usage of XSLT mappings to carry out the receiver determination and interface split pipeline steps. The idea was nicked from the extended receiver determination capability in SAP Process Orchestration where you can run an operation mapping to determine your receivers in a content-based router or recipient list pattern. On the one hand, this dramatically simplifies the integration flow models of the receiver determination and interface split. Instead of a combination of multicast branches and multiple routers for all your content-based router xpath expressions which easily blows up your integration flow model, we simply run the xpath expressions within an XSLT mapping. So, the model becomes very neat and better readable. On the other side, we prefer XSLT over graphical message mapping because XSLT mappings are supported in the Partner Directory. So, the usage of XSLT in combination with the Partner Directory also contributes to the reduction of the number of required JMS queues.

If you like to learn more about the pipeline concept, check out the Pipeline Concept chapter as part of the migration guide for SAP Process Orchestration.

If you like to apply the pipeline concept, you find the integration package at the SAP Business Accelerator Hub.

Also check out my blog series where I describe how to apply the pipeline concept for a particular integration scenario. The first blog post covers a general use case with multiple receivers and multiple receiver interfaces.

And here's the complete list of blogs in the blog series: