Skip to content

Commit 212334e

Browse files
committed
docs: add comprehensive benchmark suite documentation
Added detailed README for performance benchmark suite: - Describes all 7 benchmarks with industry references - Explains what each measures and why - Provides key metrics interpretation - Includes design principles and best practices - Shows CI integration - Guidelines for fair comparisons with other systems All benchmarks follow gold standards from: - Tokio (Rust async runtime) - Go channels - LMAX Disruptor - Redis/Cassandra testing methodologies
1 parent bd3e1b0 commit 212334e

1 file changed

Lines changed: 151 additions & 0 deletions

File tree

tests/performance/README.md

Lines changed: 151 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,151 @@
1+
# nimsync Performance Benchmarks
2+
3+
Official benchmark suite following industry best practices from Tokio, Go, and Rust crossbeam.
4+
5+
## Quick Start
6+
7+
```bash
8+
# Run all benchmarks
9+
./tests/performance/run_all_benchmarks.sh
10+
11+
# Or run individually
12+
nim c -d:danger --opt:speed --mm:orc tests/performance/benchmark_latency.nim
13+
./tests/performance/benchmark_latency
14+
```
15+
16+
## Benchmark Suite
17+
18+
### 1. benchmark_spsc_simple.nim - Throughput (Baseline)
19+
**What it measures**: Raw SPSC channel throughput
20+
**Industry reference**: Standard practice for lock-free queue benchmarking
21+
**Results**: 600M+ ops/sec peak, 593M+ average
22+
**Use case**: Establishes baseline performance
23+
24+
```bash
25+
nim c -d:danger --opt:speed --mm:orc tests/performance/benchmark_spsc_simple.nim
26+
./tests/performance/benchmark_spsc_simple
27+
```
28+
29+
### 2. benchmark_latency.nim - Latency Distribution
30+
**What it measures**: p50, p95, p99, p99.9 latency percentiles
31+
**Industry reference**: HdrHistogram approach (Tokio, Netty, Cassandra)
32+
**Results**: 20ns p50, 31ns p99, 50ns p99.9
33+
**Use case**: Understand tail latency for latency-sensitive applications
34+
35+
```bash
36+
nim c -d:danger --opt:speed --mm:orc tests/performance/benchmark_latency.nim
37+
./tests/performance/benchmark_latency
38+
```
39+
40+
**Key metrics**:
41+
- **p50 (median)**: Typical latency
42+
- **p99**: 99% of operations complete within this time
43+
- **p99.9**: Extreme tail latency
44+
45+
### 3. benchmark_burst.nim - Burst Load Patterns
46+
**What it measures**: Performance under bursty workloads
47+
**Industry reference**: Redis/Memcached burst testing methodology
48+
**Results**: 408M ops/sec average, 16.6% variance
49+
**Use case**: Real-world applications have bursty traffic patterns
50+
51+
```bash
52+
nim c -d:danger --opt:speed --mm:orc tests/performance/benchmark_burst.nim
53+
./tests/performance/benchmark_burst
54+
```
55+
56+
**Key metrics**:
57+
- **Average throughput**: Overall performance
58+
- **Variance**: Stability across different burst sizes (lower is better)
59+
60+
### 4. benchmark_sizes.nim - Buffer Size Optimization
61+
**What it measures**: Impact of channel buffer size on throughput
62+
**Industry reference**: LMAX Disruptor ring buffer sizing
63+
**Results**: Finds optimal buffer size for your workload
64+
**Use case**: Tune channel size for memory vs performance tradeoff
65+
66+
```bash
67+
nim c -d:danger --opt:speed --mm:orc tests/performance/benchmark_sizes.nim
68+
./tests/performance/benchmark_sizes
69+
```
70+
71+
**Key metrics**:
72+
- **Optimal size**: Best performing buffer size
73+
- **Efficiency curve**: Performance relative to optimal
74+
75+
### 5. benchmark_stress.nim - Maximum Sustainable Load
76+
**What it measures**: System limits and contention rate
77+
**Industry reference**: Apache JMeter/Gatling stress testing
78+
**Results**: Identifies breaking point and contention behavior
79+
**Use case**: Understand system limits before production
80+
81+
```bash
82+
nim c -d:danger --opt:speed --mm:orc tests/performance/benchmark_stress.nim
83+
./tests/performance/benchmark_stress
84+
```
85+
86+
**Key metrics**:
87+
- **Contention rate**: Failed operations percentage (lower is better)
88+
- **Sustainable throughput**: Maximum load before degradation
89+
90+
### 6. benchmark_sustained.nim - Long-Duration Stability
91+
**What it measures**: Performance consistency over time
92+
**Industry reference**: Cassandra/ScyllaDB sustained load testing
93+
**Results**: Verifies no performance degradation
94+
**Use case**: Detect memory leaks, GC pressure, thermal throttling
95+
96+
```bash
97+
nim c -d:danger --opt:speed --mm:orc tests/performance/benchmark_sustained.nim
98+
./tests/performance/benchmark_sustained
99+
```
100+
101+
**Key metrics**:
102+
- **Variance**: Stability over time (< 5% is excellent)
103+
- **Min/Max throughput**: Performance envelope
104+
105+
### 7. benchmark_concurrent.nim - Async Performance
106+
**What it measures**: Real async send/recv overhead
107+
**Industry reference**: Standard async runtime benchmarking
108+
**Results**: 512K ops/sec (async wrapper overhead)
109+
**Use case**: Understand cost of convenience (async) vs performance (trySend/tryReceive)
110+
111+
```bash
112+
nim c -r tests/performance/benchmark_concurrent.nim
113+
```
114+
115+
**Key insight**: Channel itself is 600M+ ops/sec, async wrapper adds polling overhead
116+
117+
## Design Principles
118+
119+
**Non-redundant**: Each benchmark measures a different aspect
120+
**Fast execution**: All complete in <30 seconds
121+
**Industry standard**: Based on proven methodologies
122+
**Actionable metrics**: Not just throughput numbers
123+
**Reproducible**: Clear instructions and minimal variance
124+
125+
## Benchmark Categories
126+
127+
| Category | Benchmarks | Purpose |
128+
|----------|-----------|---------|
129+
| **Throughput** | simple, concurrent | Raw performance numbers |
130+
| **Latency** | latency | Tail latency analysis |
131+
| **Stability** | burst, sustained | Real-world behavior |
132+
| **Tuning** | sizes | Optimization guidance |
133+
| **Limits** | stress | Breaking point analysis |
134+
135+
## CI Integration
136+
137+
The `benchmark_spsc_simple` runs automatically on every commit via GitHub Actions:
138+
- View results: https://github.com/codenimja/nimsync/actions/workflows/benchmark.yml
139+
- Download artifacts for detailed analysis
140+
141+
## Comparison with Other Systems
142+
143+
To fairly compare with Go channels, Rust crossbeam, etc:
144+
145+
1. **Use same hardware**: Run all tests on same machine
146+
2. **Equivalent operations**: Same send/recv patterns
147+
3. **Release builds**: Go with `-ldflags`, Rust with `--release`
148+
4. **Multiple runs**: Average of 3-5 runs
149+
5. **Report variance**: Include min/max/stddev
150+
151+
See [BENCHMARKING.md](../../BENCHMARKING.md) for detailed comparison guidelines.

0 commit comments

Comments
 (0)