Archiving of Dynamically generated Queues

[sunit_s]

Hi All.

I want to generate dynamic queue names according to a field in xml payload of message. I don't want to use one queue name for all incoming messages by using queue name in sender channel (EOIO). This blog highlighting module bean will help me to achieve it:

https://blogs.sap.com/2016/08/15/setting-dynamic-queue-name-in-eoio-scenarios/

My issue is over life cycle of these created queues. Over a period of time there will be thousands of queue names. Are the created queues deleted/archived by PO jobs in some point of time? Or some manual activity is required? Will many queues cause performance issues if not deleted or archived?

Want to understand life-cycle of dynamically generated and static queues in general.

Thanks

Sunny

[vadimklimov · Accepted solution]

Hi Sunit,

I've just come across this question of yours, while I had already answered almost all questions you raised here, in comments to the blog you mentioned. Shortly speaking, the major performance implication might be not because of number of queues or discrete sequence IDs used within Adapter Engine (as those are only attributes of processed messages, not separate physical entities), but number of holding EOIO messages that can become accumulated in queues, should there be massive number of preceding failed messages or high backlog caused by overall shortage of throughput of Adapter Engine or components it depends on - those holding messages can impact Adapter Engine startup time (although there were improvements in this area and negative impact of large number of holding messages to overall startup time of Adapter Engine has been reduced).

Regarding dynamically determined and fixed queue names... There is no difference in the way how Adapter Engine in general and Sequencer service in particular make use of such queues. They only differ in the way how queue name (or, to be more precise, sequence ID) has been determined when constructing the message - fixed at configuration time or determined at runtime - but not in the way how queues in general will be handled further by Messaging System of Adapter Engine.

Regards,

Vadim

[sunit_s · Accepted solution]

Thanks for reply Vadim.

If I understand you correctly then there wont be any performance issue if there are no large number of holding EOIO messages. That's good.

I came across below monitor that shows all the queues that are active:

http://<server>:<port>/MessagingSystem/monitor/sequenceStatus.jsp

The monitor points to Sequence Ids (queue name) and their statuses like 'ACTIVE' in table SAP<SID>DB.BC_MSG_SEQ_STAT

Just to clarify we can generate 1-5 millions of Sequence IDs with 'Active' status in this table. Can it cause any performance problem in future? E.g. if we generating 1 queue per customer, eventually there will be millions of active queues. Have you generated millions of Sequence IDs in your project and did it work fine?

Regards

Sunit

[vadimklimov · Accepted solution]

Hi Sunit,

Sequence Status Monitor that you mentioned (and underlying database table BC_MSG_SEQ_STAT) is used for statistics. Precisely, table BC_MSG_SEQ_STAT stores information on all sequence IDs that have ever been used by Adapter Engine (not only active sequences, but any sequence that has ever been used by Adapter Engine). In normal circumstances, you (as a customer) is not supposed to interfere into that table (for example, by manually cleaning its content) or terminate sequences (from Sequence Status Monitor), since termination of the sequence will have consequences to corresponding EOIO messages that will be assigned to that sequence - they will be rejected.

From my personal experience, I have never faced performance issues because of that table. Moreover, it is very common practice to generate queues / sequence IDs based on primary keys of transmitted business objects (such as customer number, material number, employee number, order number, etc.) so that sequence for each entity of that object is respected, but blockage of the queue due to an error in processing of a message for one of entities will not cause delay in processing / holding for other entities (such as, if message for customer A got error, only sequence for that customer will be put on hold, and messages for customer B will be processed further).

I understand your concerns about potential growth of the table BC_MSG_SEQ_STAT, and I recall there were some discussions about it in the past, but I haven't seen any explicit cleanup tool / technique for it so far. Although, implicitly, standard deletion job takes care of removal of obsolete entries from that table (old entries that are marked as terminated).

Regards,

Vadim

[sunit_s · Accepted solution]

Hi Vadim

Thanks for your reply

I was advised by SAP OSS that 'Default Delete Job in Background Job Processing Monitor' will delete active sequences that are not used for more than 30 days by default. No problem of creating dynamic queues. Many thanks for your reply.

Regards

Sunit