A test harness for characterising and validating the message throughput of Solace brokers — both software brokers and hardware appliances. Uses Ansible to deploy sdkperf_c to remote Linux test hosts and drive publisher/consumer load against the broker under test.
- A Solace software broker or hardware appliance to test
- Publisher and consumer test hosts (Linux) — min 2, ideally 4 or more with 10 GbE connectivity
- A controller host (Linux) with Ansible installed and SSH access to the test hosts
- SSH keys from the controller installed on all test hosts
- A client username on the broker VPN with publish, subscribe, and guaranteed endpoint create permissions (credentials stored in
config/credentials.yaml) sdkperf_cbinary placed inpubSubTools/on the controller (copied to test hosts by Ansible)
For minimal testing a single publisher host and a single consumer host is sufficient, but will limit the maximum achievable rates (especially for direct messaging at small message sizes).
Run ./setup.sh for a guided walkthrough of the above requirements and to configure your test hosts and credentials.
setup.sh # Interactive setup wizard — configures hosts and explains requirements
start-benchmarking-test.sh # Interactive menu to select and run a benchmarking test
start-standard-discovery-test.sh # Wrapper to run a generic discovery test (prompts for all parameters)
start-custom-discovery-test.sh # Builds a custom discovery testset and saves it to custom-sets/
VERSION # Harness version and release date (sourced by runner scripts)
bump-version.sh # Updates VERSION to a new semver and today's date
engine/ # Core test engine
engine/run-testset.sh # Runs a fixed-target testset (pass/fail against known rates)
engine/run-binsearch-testset.sh # Discovers max throughput via exponential probe + binary search
engine/run-test.sh # Single-test wrapper around the Ansible playbook
engine/start-sdk.yaml # Ansible playbook: deploys sdkperf_c, runs publishers and consumers
engine/analyse-result-set.sh # Parses result files and prints diagnostic guidance
benchmarking-tests/ # Fixed-target testsets for known broker tiers
discovery-tests/ # Discovery testsets (binary search format)
custom-sets/ # User-generated custom discovery testsets (gitignored)
scripts/ # sdkpublisher.sh and sdkconsumers.sh — run on test hosts
pubSubTools/ # sdkperf_c binary and licences (not included in repo)
config/host # Ansible inventory (publisher and consumer hosts)
config/credentials.yaml # Broker credentials for sdkperf (gitignored — not committed)
config/credentials.yaml.example # Credentials template
docs/ # Architecture overview and additional documentation
results/ # Test result output files
temp/ # Temporary per-iteration logs (cleaned up after each run)
Run the setup wizard to configure your test hosts:
./setup.shThis will explain the infrastructure requirements, guide you through SSH key setup, and write your publisher/subscriber host names to config/host and your broker credentials to config/credentials.yaml.
The benchmarking-tests/ folder contains pre-configured testsets for common broker tiers and configurations. Each script defines target rates the broker is expected to achieve and reports pass/fail for each scenario.
The easiest way to run one is via the interactive menu:
./start-benchmarking-test.shOr invoke a testset script directly:
benchmarking-tests/ent-10k-gm-ha.sh <broker-ip>| Script prefix | Licensing tier |
|---|---|
standard-* |
Standard |
ent-1k-* |
Enterprise 1k |
ent-10k-* |
Enterprise 10k |
ent-100k-* |
Enterprise 100k |
Each tier has variants for message type and HA configuration:
-direct— direct messaging-gm-noha— guaranteed (persistent) messaging, standalone broker-gm-ha— guaranteed messaging, HA pair (primary + backup + monitoring node (in case of software))-quick— abbreviated run covering key scenarios only
| Script | Description |
|---|---|
3560-ADB4-direct.sh |
Solace 3560 appliance — direct messaging |
3560-ADB4-gm-ha.sh |
Solace 3560 appliance — guaranteed messaging (HA pair) |
Target rates in the 3560 testsets are based on published Solace specifications (11M/24M msg/s direct at 100B f=1/f=10; 640k/2.8M msg/s persistent at 1KB f=1/f=10) and measurements from londonlab.
Use the discovery tests when you don't know what rates to expect — for example, new hardware, a new configuration, or initial characterisation of a broker.
Rather than specifying target rates, the script automatically finds the highest message rate the broker can sustain end-to-end for each scenario using two phases:
- Exponential probe — doubles the rate from a low starting point until the first failure, quickly narrowing the search window without wasting iterations in the wrong part of the range.
- Binary search — converges on the maximum stable rate within the window found by the probe.
The search stops early once precision reaches ±1% of the current midpoint (with an absolute floor of ±500 msgs/sec for very low-rate scenarios).
Test entries omit the target rate field:
msg_size:fanout:publisher_hosts:msg_type
Standard discovery — standard scenario matrix (100B, 1KB, 20KB × fanout 1/5/50), prompts for broker, SSH user, host count, and message types:
./start-standard-discovery-test.shCustom discovery — interactive wizard to choose message types, sizes, fanout values and upper bounds, generates a reusable testset under custom-sets/:
./start-custom-discovery-test.shThe generated script is saved to custom-sets/<name>.sh and can be re-run directly at any time:
./custom-sets/<name>.sh [broker-ip]The exponential probe starts at upper_bound / 1024 and doubles upward. The defaults are conservative (software broker limits). Testsets for more capable brokers should override these via export before calling engine/run-binsearch-testset.sh:
| Variable | Default | 3560 w/ ADB4 |
|---|---|---|
search_upper_bound_direct |
5,000,000 | 25,000,000 |
search_upper_bound_nonpersistent |
2,000,000 | 20,000,000 |
search_upper_bound_persistent |
1,000,000 | 5,000,000 |
See discovery-tests/londonlab-discovery.sh for an example of how to override these.
- The testset script defines scenarios as arrays and passes them to
engine/run-testset.shorengine/run-binsearch-testset.sh. - For each scenario the runner calls
engine/run-test.sh, which invokes the Ansible playbookengine/start-sdk.yaml. - Ansible copies
sdkperf_cand the publisher/consumer scripts to the test hosts, then launches:- Consumers asynchronously (started first, so they are ready before publishers)
- Publishers at the target rate for the configured
runlength(default: 60 seconds)
- After the run, Ansible collects stdout from both sides. The total consumer rate is summed across all hosts and checked against the target (allowing a 5% error margin).
- Results are written to
results/at the end of each testset, followed by an automated diagnostic analysis (see Analysing results below).
msg_size : fanout : overall_msg_rate : parallel_hosts : msg_type
overall_msg_rate is the total consumer rate expected (i.e. it already includes the fanout multiplier). The playbook divides it by parallel_hosts × fanout to derive the per-publisher-host rate.
msg_size : fanout : parallel_hosts : msg_type
The target rate field is omitted — the script determines it automatically.
engine/analyse-result-set.sh is run automatically after each testset completes. It parses the result file and prints a diagnostic summary identifying common bottlenecks and configuration issues.
You can also run it manually at any time:
# By test-set shorthand name (resolves to results/<name>_result.txt)
engine/analyse-result-set.sh 1k_mixed
# By full file path
engine/analyse-result-set.sh results/1k_mixed_result.txt
# On a directory
engine/analyse-result-set.sh results/my-run/
# Scan the whole results/ directory (no arguments)
engine/analyse-result-set.shThe script checks for:
| Finding | What it means |
|---|---|
"Error in clock" errors |
Publisher host CPU saturated — sdkperf cannot sustain the requested rate |
| Publish rate flat across all scenarios | Publisher host or NIC is the bottleneck regardless of target |
| Low publish rate consistent across message sizes | WAN or throttled network link between publisher and broker |
| Publisher rate consistently ~50%/33%/25% of expected | Fewer publisher hosts contributed than specified — one or more [pubhost] entries were unreachable during the Ansible run |
| NIC approaching 1 GbE / 10 GbE | Publisher or consumer NIC bandwidth limit |
| Low persistent publish rate (target >> achieved) | Storage IOPS limit on the broker |
| Consumer rate < publish_rate × fanout | Messages dropped or not delivered to all subscribers |
| All persistent fail, direct OK | VPN message spool quota, guaranteed messaging not enabled, or missing endpoint-create permission |
| Publisher near target but all tests fail | Broker CPU, memory, or NIC is the bottleneck |
| Message size | Bottleneck | Notes |
|---|---|---|
| ≤ 1KB | Broker CPU / disk IOPS | Throughput largely independent of message size |
| 1KB–20KB | Disk write bandwidth (persistent) or broker CPU (direct) | Persistent rates fall as message size increases |
| ≥ 20KB | Network bandwidth (n * 10 GbE ≈ n * 1.25 GB/s per host / network interface) | Rate ≈ n * 1.25 GB/s ÷ msg_size per host, fanout has minimal effect on publisher rate |
For high-fanout scenarios (f ≥ 10), the consumer-side network becomes the binding constraint: each consumer host must handle publish_rate × fanout messages. With 1 consumer host on 10 GbE this limits useful fanout testing. Use multiple consumer hosts for accurate high-fanout results.
Written by ./setup.sh and gitignored. Required fields:
| Field | Description |
|---|---|
broker_vpn |
Broker VPN name |
broker_username |
Client username sdkperf connects as |
broker_password |
Client password |
sshuser |
SSH user on test hosts |
pub_cores |
CPU cores on publisher hosts (sets parallel publisher processes) |
sub_cores |
CPU cores on subscriber hosts |
The runner scripts (run-binsearch-testset.sh, run-testset.sh) validate that the three broker credential fields are present before starting any tests and abort with a clear message if any are missing. Copy config/credentials.yaml.example as a starting point if you are not using setup.sh.
Key parameters in engine/run-binsearch-testset.sh:
| Parameter | Default | Description |
|---|---|---|
runlength |
60 | Seconds per test run |
search_iterations |
10 | Maximum binary search iterations |
allowed_error_margin |
5 | Consumer rate must be ≥ (100 − margin)% of target to pass |
precision_pct |
1 | Stop binary search when range ≤ ±1% of midpoint |
precision_threshold |
500 | Absolute minimum precision floor (msgs/sec) |
inter_iteration_cooldown |
5 | Seconds between iterations (allows broker queues to drain) |
Key parameters in engine/start-sdk.yaml:
| Parameter | Default | Description |
|---|---|---|
sshuser |
perfharness |
SSH user on test hosts |
sdk_publishers |
4 | sdkperf_c publisher processes per host (match to core count) |
runlength |
120 | Default run length (overridden by calling script) |
The harness version and release date are stored in VERSION at the repo root and sourced by the runner scripts. The version is written into the "Test environment" header of every result file, making it easy to reproduce or compare runs.
To bump the version before committing a significant change:
./bump-version.sh v2.2.0This updates VERSION to the new semver and sets the date to today. The VERSION file should be committed together with the change it describes.
docs/Perf Test Harness-Overview.pptx— architecture and methodology overview
Christian Holtfurth