What's new in the Perftest Harness — four months of improvements

General Discussions
,
1

Back in April, Hari posted about the Benchmark your Solace broker with the PerfTest Harness — an Ansible-based tool that uses sdkperf to automatically benchmark a Solace broker’s maximum throughput. Since then We’ve shipped a lot of improvements based on real-world use and customer feedback across software brokers, hardware appliances, and Solace Cloud.

Here’s a summary of what’s new.

  • Automated results analysis and bottleneck diagnosis
    • After every testset, the harness now automatically runs analyse-result-set.sh — a post-processor that parses the raw result files and diagnoses what actually limited throughput. Rather than leaving you to interpret numbers manually
  • Broader benchmarking coverage
    • The initial release shipped benchmarking scripts for standard software broker tiers. We’ve since added:
      • hiperf-* scripts — targets calibrated for high-performance on-premises servers (25 GbE+, NVMe storage), which significantly outperform cloud VM tiers
      • cloud-2025-TLS-* scripts — targets based on Solace’s own published Cloud performance figures (TLS+HA), useful for reproducing or comparing against vendor numbers
      • 3560-ADB4-GM650k-* scripts — targets for the 3560 hardware appliance with a GM650 key, covering direct and guaranteed messaging
    • All benchmarking scripts now support TLS (broker_tls: true in credentials.yaml) and will abort immediately if any scenario fails, rather than continuing with a broken configuration.
  • Mesh throughput testing — VPN bridges, MNR, and DMR
    • A new test mode lets you characterise the throughput ceiling of an inter-broker link rather than a single broker. Publishers connect to one broker, subscribers to a second, and messages traverse the configured VPN bridge, MNR, or DMR link. The harness manages both sides automatically using the same binary-search discovery engine, and the results table includes a bandwidth column so you can see both message rate and byte rate at once.
  • SEMP pre-flight checks and automated broker setup
    • One of the most common stumbling blocks was brokers that appeared to accept connections but silently blocked test traffic due to ACL or client-profile misconfiguration. The harness can now automatically check for these conditions or even set up the right profiles if provided with SEMP admin credentials with the right level of access.
  • Multi-protocol support — MQTT and AMQP 1.0
    • The harness now supports testing over MQTT (QoS 0, 1, and 2) and AMQP 1.0 alongside the existing SMF support. Set broker_protocol: mqtt or broker_protocol: amqp in credentials.yaml — everything else (discovery, benchmarking, analysis) works the same way.
  • Setup and usability improvements
    • Configurable SSH port — ssh_port in credentials.yaml, useful for non-standard jump hosts or port-forwarded environments.
    • Configurable SMF port — broker_port (and per-side overrides pub_broker_port / sub_broker_port for mesh) for brokers on non-default ports.
    • localhost auto-detection — if a publisher or consumer host resolves to localhost, the harness automatically sets ansible_connection=local and warns you, rather than silently failing SSH.
    • Dependency checks — setup.sh now checks for ansible (hard fail) and dnsutils (warning) before starting, with clear install instructions.
    • Versioning — every result file now includes the harness version and date in its header, making it straightforward to correlate results across time and releases.
  • LLM tool definitions
    • For those experimenting with AI-assisted capacity planning or automated testing workflows, docs/tools.json contains Anthropic-format tool definitions for the harness operations: run a benchmarking test, run a discovery test, run a custom testset, analyse results, list results. Drop the tools array into any Claude API call and an LLM can drive the harness programmatically.

The repo is at GitHub - SolaceLabs/solace-perftest-harness · GitHub .

As always, issues and PRs are welcome — and if you run it against your own infrastructure, we’d love to hear what you find.

1 Like