Decouple Sender and Flows for Asynchronous Transfers - Scaling Issues
Dear Cloud Integration Experts,
While I am aware of the concepts for decoupling sender and flows using JMS queues or data stores described in the link below, I am struggling with applying them on a larger scale.
Here is my scenario: We are sending a lot of asynchronous ABAP proxy messages from our SAP ERP system to Integration Suite, from where we send the messages to various receiver systems. The messages are received in Integration Suite by an XI sender adapter with temporary storage option “JMS Queue”.
Now suppose there are messages in the JMS queue for two different receivers (receiver A and receiver B) and two different interfaces for each receiver (interfaces A1 + A2, and B1 + B2 respectively).
Initial approach: Direct processing from XI sender queue
Messages are consumed from the JMS queue of the XI sender and are processed and sent to the respective receiver interfaces without further decoupling.
The problem with this approach is that there is no way to prioritize which messages should be consumed from the JMS queue. If a lot of messages for interface A1 are put into the queue at once, and it takes the receiver system a long time to return a response for each A1 transfer, all the messages for interface A2 and for receiver B that have been added to queue in the meantime are stuck until it is their turn to be consumed. With this approach one interface is basically able to block the queue for all other interfaces.
Option 1: Add JMS queues for each receiver system
Instead of sending messages to the receiver system directly after consuming them from the XI sender queue, we create one additional JMS queue for each receiver system and send the messages to these queues for decoupling. In my example, we would create JMS queues “A” and “B”.
Now we put a lot of A1 messages into the XI sender queue again. These will be forwarded to the “A” queue quickly and no longer block the XI sender queue for messages to receiver B. However, while messages for interface A2 will also be forwarded quickly from the XI sender queue, they will get stuck in the “A” queue behind all the A1 messages.
So if we rely on a having a near-real-time transfer for specific interfaces, this approach will not solve all our issues.
Option 2: Add JMS queues for each interface
Instead of creating JMS queues for each receiver system, we create one JMS queue for each interface. In my example, we would create four JMS queues “A1”, “A2”, “B1”, “B2”.
While this approach will solve the issues described above, it will not scale at all unfortunately, as the number of JMS queues is limited and we will quickly run out of JMS queues when adding additional interfaces.
Option 3: Add Data Stores for each interface
Instead of forwarding the messages from the XI sender queues to other JMS queues, we will forward them to Data Stores instead. There is no hard limit to the number of Data Stores, so this looks like an approach that can potentially scale very well.
But using Data Stores has one major drawback from my point of view. The Data Store sender adapter does not allow parallel processing. While a JMS sender can be configured to consume multiple messages in parallel, the Data Store sender will only pick up the next message after the previous message has been processed.
With that approach it is not possible to send multiple messages for the same interface in parallel to the receiver system. But if the receiver system supports that, we absolutely want to do that when transferring lots of messages at once to significantly shorten the total runtime in these scenarios.
In theory we could possibly forego the Data Store sender adapter, and instead select multiple messages manually from the Data Store and split them. But this looks like an awkward approach. The select would have to be triggered by a timer every few seconds, and this will congest the message monitoring. There is probably also additional development needed to ensure that only those messages are finally removed from the Data Store that were processed successfully after the split, while the error messages must retain at the data store and should be repeated with exponential backoff - an option that is already built into the sender adapter for good reason.
Summary
All in all, it looks like we must mix these different options on a case-by-case basis in our landscape. But the decision must be made at design time for an interface and there is no way to react to dynamically changing message volumes.
- Use JMS queues, but limit the number of queues while at the same time somehow make sure that one mass transfer will not block your other interfaces - which it seems is only possible by running these through dedicated queues.
- Use Data Stores, but live with the fact that parallel transfers are not possible.
Are there any best practices for handling these scenarios, or are there other concepts we could apply?
Kind regards,
Nils