Python Message Transfer App (DMQ Reprocessing)

Connectors & Integrations
, , ,
1

Hi Solace Community,

I’m working on a little project related to Dead-Message-Queue processing. Part of this is monitoring, alerting and investigating messages that arrive in a DMQ. The last part is reprocessing (where appropriate) the messages back via the input queue for consumption. The first part of the project is all using python so ideally the last part I’m trying to complete using the solace.messaging package — Solace Messaging API For Python 2025 documentation

Previously for this same task we have used various methods: JMS Toolbox (doesn’t scale), external client apps (overkill) and also python script using SEMP copy-message feature. The copy-message feature works well for small workloads but is slow when we are dealing with huge DMQs.

I’ve been looking at available sample repos that use the PubSub Python API including: https://github.com/SolaceSamples/solace-samples-python

I’ve been able to create a python Queue-Queue transfer app however having a couple issues:

  • Using persistent PubSub I can only seem to transfer between Queue-Topic, rather Queue-Queue. For one off tasks this is fine, but isn’t ideal for a utility tool where we add a temporary topic each time.

  • I’ve had trouble with increasing performance in regard to acknowledgements and window/batch sizes. When correctly acknowledging messages, I can only seem to process single messages at a time. Ideally, I would like a much larger window size to process 1000s of messages a minute. However, do not want to risk any message loss during message transfer.

    To give some context - I’m currently dealing with a rare case where a DMQ has 1.4M messages. This is due to a client disconnecting for a longer period from authentication issues - and messages reaching TTL and moving to the DMQ. Processing this with any of our current solutions is not feasible.

If anyone has some suggestions or a repository reference that may assist with a solution that would be greatly appreciated.

Cheers

2

Hi there, I’m Solly AI, I generate responses to your questions using the power of the Solace Agent Mesh:

Please remember that AI technology can make mistakes so ensure that you read our Disclaimer.

AI Generated responses are a work in progress, your help in evaluating my response would be awesome! If my response makes sense, please mark it as a solution or give a :+1:. If my response doesn’t make sense for your topic, please react to it with a :shaking_face:.

Let me analyze the requirements for implementing high-performance Queue-to-Queue message transfer from a DMQ using the Python API.

Reasoning:

  1. The scenario involves consuming messages from a DMQ (Dead Message Queue) and requires:

    • High performance processing
    • Large volume (1.4M messages)
    • Proper acknowledgment
    • Window/batch sizing considerations
    • Using Python API
    • Software broker

  2. From the documentation:

    • DMQs are regular durable queues that can be consumed like any other queue
    • The Python API supports consuming from queues using PersistentMessageReceiver
    • Messages must be acknowledged to be removed from the queue
    • Back-pressure handling is important for high-volume scenarios
    • The API supports both synchronous and asynchronous consumption patterns

  3. For high performance with large volumes:

    • Asynchronous consumption would be more efficient than synchronous
    • Proper acknowledgment handling is critical
    • Need to consider back-pressure to handle the large volume

Summary Answer:
To implement high-performance Queue-to-Queue message transfer from a DMQ using the Python API:

  1. Create a PersistentMessageReceiver to consume from the DMQ:

    • Use MessagingService.create_persistent_message_receiver_builder()
    • Configure it for the DMQ queue
    • Start the receiver

  2. Implement asynchronous message handling using MessageHandler:

    • Process messages in the on_message callback
    • Implement proper acknowledgment using receiver.ack()
    • Consider batching acknowledgments for better performance

  3. Handle back-pressure appropriately:

    • Monitor internal buffer capacity
    • Implement appropriate back-pressure strategy
    • Consider using pause() and resume() for flow control if needed

For specific implementation details and best practices, refer to the Solace documentation:

3

Allen,

Firstly I would be asking why messages are going to the DMQ. In my thinking it only makes sense to route messages to DMQ where bad data is involved, else you should be handling and retrying events for technical errors. No point routing millions of messages to DMQ if destination is unreachable.