Awesome @harry . Looking forward to seeing the results!
A couple thoughts… I don’t know Spark, but I do know that creating new Connections and Producers is kind of an “expensive” operation (vs. sending a single message is “cheap”), so hopefully those objects are somewhat long-lived, and you’re not creating and destroying Connections constantly… more efficient for the connection to remain up as long as possible, and just send the rows as messages as you iterate over. I think that sounds like what you’re doing, but just wanted to be clear.
Are you sending to a dynamic topic? E.g. pulling out some piece of data from the row (id?) and inserting that into the topic that you’re publishing on? E.g. spark/data/$id/update or something…?