on 09-24-2021 12:44 AM
Hi,
I am trying to get my CAP application working with Event Mesh and Dead Message Queues, but I cannot find the appropriate way to reject an inflight message so it gets back onto the queue and eventually moves to the DMQ.
I have this in my cds config:
"cds": {
"requires": {
"messaging": {
"kind": "enterprise-messaging-shared",
"queue": {
"name": "default/my/queue",
"maxDeliveredUnackedMsgsPerFlow": 5,
"maxRedeliveryCount": 10,
"deadMsgQueue": "default/my/queue/dmq",
"maxTtl": 0,
"respectTtl": true
}
},
...
and sample code like this:
const cds = require('@sap/cds');
module.exports = async srv => {
const messaging = await cds.connect.to("messaging");
messaging.on("default/my/topic", (msg) => {
return new Promise((resolve, reject) => {
setTimeout(() => {
reject();
}, 1000);
})
});
}
I.e it returns a promise which gets rejected after a second. I was expecting that the failing the message would cause the error to propagate back to the queue, get retried until the TTL or Max Count expires and then move to the DMQ. But the actual result is that the message simply gets consumed.
If I throw an error from within the handler instead, the message fails and gets retried indefinately.
const cds = require('@sap/cds');
module.exports = async srv => {
const messaging = await cds.connect.to("messaging");
messaging.on("default/my/topic", (msg) => {
throw new Error("!");
});
}
Can somebody shed some light on how this should be implemented?
Thanks!
//Carl
Hi onnheimc and david.kunz2,
I just did a quick test with the plain node.js AMQP lib. I just slightly changed the included consumer example:
This is working as expected. I get the message a total of 3 times, as I have configured 2 redeliveries on my queue settings.
Not sure how this translates to the CAP implementation.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi tobias.griebe , david.kunz2 ,
This turned out to be caused by the way failed messages were rejected by CAP into the Event Mesh. Maybe that is what David meant in the last comment above. It used to be returned with a "failed" but was corrected to be returned with a "rejected" in the february release of CAP I think.
So this works fine now. I still have some issues with processing the DMQ however. In particular I am struggling to find the right strategy to handle the dead messages once the root cause of the error has been resolved. Our scenario is such that all messages must be processed (or actively discarded). It does not need to happen in order. So when a message fails, we move it to the dmq and operators monitor that, solve the root case (usually bad master data) and then needs to reprocess the messages. I am not finding a smooth way of doing this however. We sometimes get surges of failures (hundreds of thousands with single root causes).
Currently we are using a batch job which I have placed inside the CAP application to move the messages back. We have the job scheduler call on it, and then it consumes from the dmq, posts back to the original topic and finally confirms the consumption - using the REST protocol. It works, but has a number of flaws. The approach seems wrong but I do not see good alternatives. In the end I want CAP's "on" handler to run, but I cannot see a way to consume the DMQ straight into it. I also do not see a way to shovel blocks of DMQ messages back onto the original queue.
Would love to hear your thoughts on how to approach that scenario. It is a bit off topic for this question though, maybe this one instead.
Thanks!
//Carl
Hi onnheimc ,
Exactly, there ware some issues in the underlying message infrastructure which prevented the counter to work when failed() was called without an error object, that has been changed in CAP release.
Regarding consumption of messages in the DMQ, this is unfortunately not supported in CAP at the moment.
In principle, you could also use the AMQP library directly: https://www.npmjs.com/package/@sap/xb-msg-amqp-v100
Best regards,
David
User | Count |
---|---|
101 | |
13 | |
13 | |
11 | |
11 | |
7 | |
6 | |
5 | |
4 | |
4 |
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.