I have a question regarding best practices for queue design in Solace / SAP AEM.
Considering that a queue can subscribe to multiple topics, what is the recommended approach for designing queues for target systems/consumers?
Option 1:
Create a single queue per target system and subscribe this queue to multiple topics.
Example:
salesforce.queue
sap/s4/customer/*
sap/s4/salesorder/*
sap/s4/billing/*
Option 2:
Create separate queues per business domain/scenario, such as:
salesforce.customer.queue
salesforce.salesorder.queue
salesforce.billing.queue
In this case, the target system would need to connect to multiple queues.
I already worked on a project where we initially separated queues by system and scenario, but we quickly exhausted the broker connection limit (250 connections). Because of that, we had to redesign the architecture and centralize everything into a single queue, letting the consumer analyze the received topic and handle the processing logic internally.
This raised a question for me about what is considered the best practice in real enterprise scenarios.
From a best practice and operational perspective, which approach is usually recommended in Solace / SAP AEM enterprise architectures?
First, a question: do you have FIFO requirements for these topics? This single requirement dictates whether your queues must be defined as Exclusive or Non-Exclusive, which completely impacts the architectural approach.
If you go with Option 1 to save broker connections and need FIFO, the queue must be Exclusive. This may create a performance bottleneck because only a single active consumer can process messages concurrently across all combined topics. On the other hand, Option 2 provides much better fault isolation per business domain but, as you experienced, you will exhaust the 250 connection limit much faster.
Finally, if ordering is crucial, have you considered using Partitioned Queues to balance strict message ordering with horizontal scalability? The trick here could be using the “business domain” (e.g., customer, salesorder) as your partition key. This guarantees FIFO within each specific domain while still allowing multiple consumer instances to process the queue concurrently.
I am also interested in this topic, and this is an idea I have been wondering about, but I haven’t looked into in detail.
Our current design matches “option 2” in the OP, and perhaps it’s worth spending a few words explaining why: We are using solace for Event Carried State Transfer between systems (not request/reply). In other words, our main use case is as a near-real-time replacement for traditional scheduled ETL processes.
For example, we might have a system of record for all of our sales orders. But the delivery system, which might be some third party application, also needs to know about sales orders. We have to populate the “orders” table in that external system’s database, so we copy sales orders over to the delivery system. But a traditional bulk ETL (eg, via SSIS or Azure Data Factory) introduces too much latency.
So we set up a publication for a SalesOrders topic. When a subscribing system wants to know about sales orders, we will create a queue for them, subscribing to that topic. For example, the queue DeliverySystem/SalesOrders, subscribed to the topic SalesOrders. As sales orders change the messages must be delivered in order on that topic, but if you also subscribe to Customers, we don’t guarantee anything about order across different topics. That’s not necessary (and in fact impossible for various reasons).
Note that we are not creating a subscription for some particular sales order in order to execute a business process bound to that order. And the topic we are publishing doesn’t include a sales order id in the path (eg SalesOrders/123). We publish all of the sales orders to the SalesOrders topic. I don’t see any benefit in adding order id’s, customer id’s, etc, to the topic.
OK, so I am looking ahead and have the same concern about option 2. If we have, say, a dozen different applications, and a hundred or so different entities to publish (ie, topics), and each system wants a couple of dozen entities, then that’s a lot of subscriber queues to create. In production we will last for a while on an enterprise 1k, but even that could, eventually, end up needing too many queues.
Creating a single queue per destination application would basically solve this problem completely, and using the entity names as partition keys fulfils the need to keep all messages about that entity type in order.
But is this a reasonable approach? Don’t you have to specify the partition count up front? Isn’t partitioning designed more for scalability, not intra-queue segregation of different messages types? If the subscriber wanted a “different flow” for each type, wouldn’t they have to be able to specify the partition they want to read from (and isn’t that impossible in the API?). Does it matter that volume in each partition would be inherently incapable of being balanced, since the partition key would explicitly not be designed for balance, it’s basesd on semantics (the message type)?
In short, is the partitioned queue approach actually a good idea here, or just a possible (but ugly) hack?
Hey @allmhhuran , @jrhuerga and @zgl : This is a great discussion and really focuses on a lot of architectural topics in EDA. Let me put up my point of view both on the original question and the follow-up comments.
The TL;DR version first
For your usecase (event carried state transfer, near real-time ETL replacement, option 2 : seperate queues per domain would be architecturally correct. But the key would be to combine it with session multiplexing and where ordering matters : partitioned queues keyed by entity id.
The long version :
One point to make clear first : the Solace (or SAP AEM) broker stores only a single copy of each message in the spool even if multiple queues subscribe to the same topic.
Having multiple queues for each domain would be the correct choice because :
Fault isolation : A problem in the salesforce.billing.queue (poison messages, slow consumer etc) is completely self contained and does not impact others like the salesforce.customer.queue
Observability : You get per-domain message rate, backlog depth and consumer health as first-class metrices. With a single queue, you loose this granularity or will have to pay the cost someplace else.
Broker side filtering and routing: One of the key features of the Solace (SAP AEM) broker is the ability to filter and route events based on the topic and provide the consumers with only those events which they actually need. If you look at our best practices or even SAP topic definitions, you see a very descriptive topic structure for this main reason. Ignoring this feature and doing the filtering on the consumer side is a performance pitfall which should be avoided.
With respect to the connection limit problem, while it is a constraint its not necessary to sacrifice your queue granularity. You can use session level multiplexing i.e. a single client session/connection can bind to multiple queues simultaneously. This means you can use micro-integrations like the Salesforce target micro-integration to connect upto 5 different queues to different destinations in Salesforce with a single connection.
For the questions around partitioned queues :
Do you have to specify partition count up front?
Yes, the max partition count is indeed setup at queue creation time. Updating it later will require the queue to be drained first.
Is partitioning for scalability or intra-queue segregation?
Scalability with in-context ordering is main target for partitioned queues. A partitioned queue is not designed to route different message types to different designated consumers. Consumers bind to the queue and receive whichever partition(s) the broker assigns them; they cannot request a specific partition.
Is the “partition key = business domain” idea a good one for this use case?
No, not really. If you use customer, salesorder and billing as the partition keys, you will get 3 logical partitions with very semantically unbalanced loads. The broker distributes partitions and not messages evenly, so this will not give you balanced throwput.
Thanks Hari. Yes, the architectural choice of “one queue per subscriber per entity” just seemed “obviously correct” and in fact I didn’t really consider any other option when designing our implementation. I only started worrying about connections when we started really pushing up our topic count and I “did the math” to figure out how long we would last in our dev environment (which is not a 1k instance! ).
Regarding:
If you look at our best practices or even SAP topic definitions, you see a very descriptive topic structure for this main reason. Ignoring this feature and doing the filtering on the consumer side is a performance pitfall which should be avoided.
This is an interesting topic - - in its own right and I might start a new thread for it, because I have found the hierearchical nature of the topic structure to be difficult to employ for effective filtering.
In my case, I do not require FIFO across different topics.
I work with SAP, and until today, in the integrations I have implemented between non-SAP systems and SAP systems using Solace (SAP AEM), we usually place SAP Cloud Integration in front of the broker. The reason is that many of the systems I have worked with do not have the capability to connect directly to the broker and only work through APIs, where we also frequently need field mappings and transformations.
Normally, the architecture looks like this:
SAP ERP → SAP AEM (Solace) → SAP Cloud Integration → Target Systems
Cloud Integration becomes the queue consumer.
If I have 3 different topics (customer, salesorder, billing), then I have 3 different interfaces connected to AEM. Until now, and I may be wrong here, I have not seen the connector specifically created for SAP AEM in Cloud Integration allowing all these different interfaces to share a single session/connection.
For example, if I have a sales interface, I can parallelize that same interface and reuse the same session, but when I have one sales interface, one customer interface, and one billing interface, they become 3 separate interfaces with 3 separate sessions/connections to the broker.
I also believe micro-integrations do not fully fit my use case, because many times the target systems are highly customer-specific applications.
Regarding this point:
“One of the key features of the Solace (SAP AEM) broker is the ability to filter and route events based on the topic and provide the consumers with only those events which they actually need. If you look at our best practices or even SAP topic definitions, you see a very descriptive topic structure for this main reason. Ignoring this feature and doing the filtering on the consumer side is a performance pitfall which should be avoided.”
I agree that topic definitions are well structured, however, the challenge appears when integrating with many different systems. In my current project, I have around 10 systems receiving messages from AEM.
If I create separate queues per system and also split them by business domain, and for each one I need a dedicated integration flow/interface that represents a new broker session/connection, I very quickly reach the connection limit.
What I have been doing is creating a more generic queue that receives less critical topics, connecting only a single interface to that queue, therefore consuming only one connection. I then let the middleware identify the topics and route the messages to the correct APIs.
For scenarios that are more complex, have very high volumes, or are more critical, I create dedicated queues.
In the end, this was the approach I found to help mitigate the connection limit issue.
In my case, I believe that even if I partition the queue, with one partition per scenario, having 3 different interfaces connected to the same queue would still consume 3 connections. I will run some tests to confirm this.
Hello @Hari_Rangarajan
The information you shared about connection reuse was very helpful, thank you.
I checked the documentation and saw that it is possible to pass the parameter JCSMP Properties → ADAPTER.CUSTOM_PROP.RECEIVER_CONNECTION_REUSE=true to reuse the session across the entire tenant, not per individual interface. In this case, I could have 100 interfaces and all of them would consume only 1 SMF connection on the broker.
This would definitely address my problem and I could work with queues at a finer granularity, for example: queue target_system.domain.
The only issue is that this parameter only works for adapters sending information to the broker (Receiver adapters), not for those consuming from it (Sender adapters). So if I have 20 interfaces publishing to the broker, those 20 share only 1 SMF connection, but if I have 100 interfaces consuming, all 100 will open 100 separate SMF connections.
Do you know if Solace supports a parameter like ADAPTER.CUSTOM_PROP.SENDER_CONNECTION_REUSE=true?
In my tests it did not work. I connected some microservices to the queue to consume the information and each one got its own session, even when using the same connection parameters, as the documentation states.
).
- in its own right and I might start a new thread for it, because I have found the hierearchical nature of the topic structure to be difficult to employ for effective filtering.