Advanced Event Mesh - Partitioned Queues : One Que...

BarisBuyuktanir · ‎01-30-2024

"Load Balancing and Context-Aware Auto-Scaling are effortlessly handled by the Advanced Event Mesh, enabling a performant event broker in various use cases, along with flexible configuration to reduce operational overhead. In this blog, I will showcase one such feature: Partitioned Queues."

Partitioned Queue (Source and Consumer Applications)

Advanced Event Mesh provides different capabilities for different use-cases. One of the latest features that is provided recently is the “Partitioned Queues”

As you might already know + certain blogs & articles already mentioned; there are different types of queues in AEM in terms of access type:

Exclusive queue where only one consumer (first binded) can receive a message at any one time, while additional consumers may be connected as standby. (do not process messages actively)

Non-exclusive queue, where multiple consumers can bind to the queue actively. This enables load balancing because the broker delivers messages to the consumers in a round-robin fashion.(randomly)

This mechanism is quite flexible and advantageous, particularly in cases where the order and processing of messages do not matter.

However there might be cases where you want the same kind of / related messages handled by the same application (consumer). For instance the very same order or lets say the same order type is to be handled by the same application: In a round robin fashion it’s not guaranteed as the second message is received about the order (i.e when order is updated) might be handled by another consumer which is “more” available at that time.

Another way to ensure this is by having exclusive queues, which tightly couples the consumer to a predetermined queue. This might become complicated as the number of variations increases.

For these kind of use-cases, Advanced Event Mesh has a subset of non-exclusive queue : Partitioned Queue where each partition within one-queue maps one and only one consumer application which guarantees the messages in the same partition to be handled by the same application. Depending on the number of partitions, one consumer application might be binded to multiple partitions, though.

Then let’s say based on the load, you increase the number of consumer services. Then the magic happens.. The broker re-balances the partitions with the consumer applications keeping the base principle the same “one partition binded to at most one consumer”.

Detailed information regarding queues (partitioned queues) can be found here in this link

This is “kind of” Solace’s/SAP's answer to Kafka Consumer Groups (in fact with more flexibility/easiness). The feature included in AEM comes with no extra cost and supported by different APIs and different protocols (JCSMP, JMS, CCSMP, Python, Golang, .NET, and JS APIs as well as standard REST/HTTP, MQTT 5.0, and AMQP etc.) similar to regular queues. (for variations/restrictions please check the documentation).

The differentiator here is the partition key set by the publisher causing the consumers to be organized flexibly, which behind the scenes handled by the event broker(AEM).

Via this mechanism, you have context aware load balancing with autoscaling. (Similar messages, depending on the context, are routed to the same consumer. Meanwhile, the number of consumers can increase with the number of partitions, and the broker manages the reorganization of this matching to ensure consistency.).

Enough theory, let’s have a working example reflecting these things:

SCENARIO

Imagine a simplified use case where sales orders are published to Advanced Event Mesh, where we want the same order handled by the same consumer(updates for instance) but within one queue configured and in an automated way.

For this we have initiated 3 consumer services, processing the sales orders from one queue. Then in year end the frequency of the orders increases, therefore we need more services and added one more binded to the same sales order queue. (but still want the remaining orders already processed, would be processed with the same consumer microservice.)

Assume that our order numbers are in format 90000XXXX. If I create an order with number 900004057; then make subsequent updates to the same order, I would like Consumer X to handle the messages related to that order.

For this the only thing I should do is having this ID in the Partition ID of the message, which is JMSXGroupID (if you use REST, it’s Solace-User-Property-JMSXGroupID) and make the necessary configurations like below for the queue and subscription.

STEP 1- Create the queue as partitioned queue

Create Queue

I simply named it q/par1

STEP 2- Make it non exclusive and set number of partitions to 3.

Queue Details

Count: The count of partitions of the queue. Only relevant for queues with an access type of non-exclusive. When zero, bound clients receive messages round-robin. Otherwise, bound clients receive messages from individually assigned partitions.
You can have a lot more, but keep it in mind that the idea is to align with number of consumers you have)
Rebalance Max Handoff Time is the maximum time (in seconds) to wait before handing off a partition while rebalancing.
Rebalance Delay is the delay (in seconds) before a partition rebalance is started once needed.

So the end result would be, we’ll have 3 partitions in our queue with no binding.

Partitions' Statuses

STEP 3- Assigning a subscription to the queue as usual

Subscriptions

I assigned bb/salesorder/> (for handling bb/salesorder/created, bb/salesorder/changed etc.) to route all sales order from my source system (let’s say S/4HANA) to the same queue. (q/par1)

Now the fun parts..

Publisher and the Subscribers(Consumers)

Publisher part is simulated by a REST call to the broker from Postman. (Please refer to my old blogs below for publishing to a topic via REST)

Publishing messages from Postman

As STEP 4, we need to publish multiple messages by putting the order number to the JMSXGroupID header(you can do this via a script or a publisher application generating random messages)

As STEP 5, for the Subscribers, I create 3 instances of the same subscriber application(as consumers). Thanks to Aaron Lee and Solace Team for the samples. I am using a little bit of modified version of the subscriber code provided in the code samples (Solace Labs) to demonstrate the behaviour of 3 consumers.

In fact, the code samples provided include a lot more than simple consumers and publishers; rather, they encompass many aspects related to partitioned queues and include artifacts and utilities for demo purposes. I strongly recommend that you take a closer look at the code samples in detail.

Consumer Applications binding to the Queue

Now these applications are binded to the same partitioned queue that we have created in the previous steps, they look like as below from the broker.

From AEM Perspective - Clients binded to the Queue

Now we can see that we have 3 consumers binded to 3 different partitions for our queue.

In fact I have created one more consumer but as there is no partition to bind, now it’s in stand-by position(inactive).

Consumers ( 4 for 3 partitions)

Now that we have our consumers expecting messages; as in STEP 6, I will send some messages to the broker, with different order numbers to see its effect. (See the messages in brown color)

Consumed messages by consumers 1,2,3,4

I have sent 6 messages. Consumer#1 receives 1 message, Consumer#2 receives 2, and Consumer#3 receives 3.(Interesting coincidence:) Obviously with high frequency, it will be more “normally” distributed. The last one (Consumer#4) doesn’t receive any as it is not connected to a partition (there are only 3 partitions created, remember).

Now as STEP 7, I will increase the number of partitions to 5, still keeping the number of consumers at 4; and expecting the broker to rebalance the partitions with the consumers.

Status of Partitions / Clients after 5 partitions

As you can see one consumer(ending with 53e5) is now binded to 2 partitions, while the others are binded one-to-one to the remaining partitions.

After I sent couple of more messages; below is the status of the consumer applications:

Status after new re-balancing

Now as the last STEP; I will send 2 updates to the order 900000723 and expect to see it is consumed by Consumer#1(as Consumer#1 processed the initial creation of 900000723) , and 2 updates for 900002474 and expect it to be received by the Consumer#3.

Similar order related messages are consumed by the same consumer microservice.

"Voilà!"
Updates to the created orders are handled with the same microservice as you can see from the monitor."

Although it’s a very simple demonstration of partitioned queues, I think it’s enough to prove how powerful and flexible Advanced Event Mesh is to load-balance & re-balance its clients and the partitions for handling the “context-awareness” without much effort.

You can apply the same pattern to multiple scenarios, such as tickets, order types, employee related scenarios, accounts, transactions etc and scale-up easily at the same time.

You can find great blogs and demo videos created by the Solace team and navigate to the documentation if you need more details regarding partitioned queues and its usage.

Below you can also find the previous blogs I mentioned about other very usable features of Advanced Event Mesh.

Advanced Event Mesh - Partitioned Queues : One Queue for All

SAP PI for Beginners

ABAP 7.40 Quick Reference

Fiori: technical installation and configuration of one app from A - Z