Hi! We’re using Spring Cloud Stream with Solace and we ran into a 503 Max Router Publishing Flow Exceeded error. I’m trying to understand may have gone wrong and how we should do it right.
Observed error
The logs go roughly like this when error occurs:
Creating producer to TOPIC my/topic/67 <message handler ID: xxx...>
Client-67-420..: Error Response [503 Max Router Publishing Flow Exceeded], txSessionId=-1, flowName=...
Unable to get a message producer for session JCSMPSession
com.solacesystems.jcsmp.JCSMPErrorResponseException: 503: Max Router Publishing Flow Exceeded
at com.solacesystems.jcsmp.impl.flow.PubFlowManager.doPubAssuredCtrl(PubFlowManager.java:355)
at com.solacesystems...
Stopping producer to TOPIC my/topic/67 <message handler ID: xxx...>
xxx... is not the last user, persisting producer...
Failed to create producer binding; retrying in 30 seconds
java.lang.RuntimeException: Unable to get a message producer for session JCSMPSession...
Trying to unbind 'my-binder:my/topic/59', but no binding found.
Could not send event: 64883 due to exception: Dispatcher has no subscribers
for channel 'myapp.my/topic/67'.
Broker settings
maxIngressFlowCount = 100 per client connection.
Temporary fix
Restarting the pod restores publishing but I suspect it is temporary.
Event flow
The application gets messages through functional bindings, does its thing and stores messages to send in an outbox table. We use streamBridge to send messages in the outbox table, like streamBridge.send(myDynamicTopic, myBinder, myMessage).
Suspected bandit
Topics are dynamic like my/topic/1, my/topic/2… and high cardinality, so there can be a really high number of dynamic destinations in streamBridge.send(myDynamicTopic, myBinder, myMessage), like potentially hundreds. Exact number at the time of error unknown.
- Could this lead to the exceeding max publishing flows on the Solace Broker?
- Is one producer flow per dynamic destination produced this way and expected for the Solace binder?
- Are flows released/reduced, how?
- What is a good fix for this?
Stuff details
Spring Boot 3.2.11
Spring Cloud 2023.0.3
solace-spring-cloud-bom 4.5.0
spring-cloud-stream-binder-solace 5.5.0
Java 17, Kotlin 1.9.25