Hi there @karthikv ,
Right, ok… for your first point (and your 2nd point too): yes the broker will move a TTL-expired message to the queue’s DMQ (if configured) as soon as it expires. (“as soon” == broker checks for TTL expiry once-per-second, that’s the maximum resolution, but if you have a TON of messages on your queue then the broker might be a bit slower at expiring messages to DMQ). If the message is in-flight or at the client but hasn’t been ACKed yet, doesn’t matter… it will be moved to DMQ. There’s nothing you can do in the client or the broker to prevent this. (today)
Strategies to help minimize this:
- Increase TTL of message / queue
- Process messages faster! ?
- Have the publisher set the “expiration” on the message as well to give consuming clients an indication that the message might have already moved to DMQ. The broker doesn’t action on the expiration, just the TTL.
- You can always have unintentionally slow consumers, bugs happen, there will always be some corner-case where you will end up in this situation even with the best strategies. It’s always best to code your consumers to be able to detect/handle duplicate deliveries.
- That said: the consumers listening to the DMQ should probably be very different apps than the regular consumers on the main queue… they’d be more error-handling, exception-dealing-with applications… so “duplicate processing” shouldn’t really be an issue.
Note that even without a DMQ, using a regular exclusive queue, there are ways for the primary consumer to lose connection to the broker where it doesn’t notice right away (hung socket through a proxy for example) and the app doesn’t know it has been disconnected, and all unACKed messages are sent to the next standby consumer on the queue, while the first app is unaware and just keeps processing (nothing to do with TTL or expiration). At least in this case, the broker sets the “redelivered flag” to give an indication to the next consumer that it might be a dupe, but if they process it first, and then the slow original consumer finally processes the same message…!? Yup, handling dupes is important.
I’m curious about QPID’s behaviour. Is it just that: if a message has been sent to a consumer, it won’t/can’t expire to DMQ? The broker will hold it in the queue until either the consumer ACKs/NACKs or disconnects?