How do avoid 403: Replication Is Standby ?

General Discussions
1

We are using “spring-cloud-starter-stream-solace” to consume the message from queue but receiving the below error frequently. Please refer the below details and help us to avoid the issue.

application.yml configuration:
binders:
           <>:
              type:
               environment:
                 solace:
                    java:
                       clientUsername: <>
                       connectRetries: 3
                       connectRetriesPerHost: 0
                       reconnectRetries: 3
                       host: tcps://<<Host Name 1>>:55443,tcps:// <<Host Name 2>>:55443
                       msgVpn: <>
solace:
           bindings:
              <>-in-0:
                 consumer:
                    queueAdditionalSubscriptions: <>
                    provisionDurableQueue: true
                    queueNamePrefix: ‘’
                    useFamiliarityInQueueName: false
                    useDestinationEncodingInQueueName: false
                    useGroupNameInQueueName: false

Error Message
com.solace.spring.cloud.stream.binder.inbound.InboundXMLMessageListener.receive - Received error while trying to read message from endpoint <>
com.solacesystems.jcsmp.StaleSessionException: Tried to call receive on a stopped message consumer.
             at com.solacesystems.jcsmp.impl.flow.FlowHandleImpl.throwClosedException(FlowHandleImpl.java:1957)
             at com.solacesystems.jcsmp.impl.flow.FlowHandleImpl.receive(FlowHandleImpl.java:899)
             at com.solacesystems.jcsmp.impl.flow.FlowHandleImpl.receive(FlowHandleImpl.java:866)
             at com.solace.spring.cloud.stream.binder.util.FlowReceiverContainer.receive(FlowReceiverContainer.java:279)
             at com.solace.spring.cloud.stream.binder.util.FlowReceiverContainer.receive(FlowReceiverContainer.java:211)
             at com.solace.spring.cloud.stream.binder.inbound.InboundXMLMessageListener.receive(InboundXMLMessageListener.java:93)
             at com.solace.spring.cloud.stream.binder.inbound.InboundXMLMessageListener.run(InboundXMLMessageListener.java:73)
             at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
             at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
             at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
             at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
             at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: com.solacesystems.jcsmp.JCSMPTransportException: (JCSMPTransportException) Error communicating with the router. (KeepAlive)
             at com.solacesystems.jcsmp.protocol.impl.TcpClientChannel$ClientChannelReconnect.call(TcpClientChannel.java:2438)
             … 4 common frames omitted
Caused by: com.solacesystems.jcsmp.JCSMPErrorResponseException: 403: Replication Is Standby
             at com.solacesystems.jcsmp.protocol.impl.TcpChannel.executePostOnce(TcpChannel.java:232)
             at com.solacesystems.jcsmp.protocol.impl.ChannelOpStrategyClient.performOpen(ChannelOpStrategyClient.java:97)
             at com.solacesystems.jcsmp.protocol.impl.TcpClientChannel.performOpenSingle(TcpClientChannel.java:418)
             at com.solacesystems.jcsmp.protocol.impl.TcpClientChannel.access$800(TcpClientChannel.java:114)
             at com.solacesystems.jcsmp.protocol.impl.TcpClientChannel$ClientChannelReconnect.call(TcpClientChannel.java:2274)
             … 4 common frames omitted

2

Hi @JamilAhmed , Need your support to resolve the issue.

3

Hi @vadivelan ,
I just tested in my environment with some brokers that are in a DR replication setup and have reproduced the error.
There are two parts here:
The reconnect parameters you’ve set that govern how long the services wait for the broker connection to get restored. (You’re essentially giving up too quickly.) Excessive logging when the service gives up connecting but the binder continues to do some work For point (1), the settings and resulting behaviour is the same as what @Kaliappans hit in an earlier issue. @marc.dipasquale recommended to adjust them or perhaps even take them out entirely and rely on the defaults instead: StaleSessionException - Tried to call receive on a stopped message consumer — Solace Community
For point (2), looks like this has been seen before too and is already tracked in a Jira issue: [SOL-79061] Very verbose logging when connection to broker is interuppted: · Issue #174 · SolaceProducts/solace-spring-cloud · GitHub . While previously it was reported in the context of a High-Availability switch, your case involves a switch between Replication mates. We’ll make sure to cover off all these situations in the behaviour improvement for it. (fyi: @giri , @amackenzie )
Thanks
Jamil