Skip to content

Performance Benchmarks

Jump to bottom
Chris Michael edited this page Jun 5, 2026 · 14 revisions

Meris

Performance Benchmarks

benchmarks/
  jmh/
  workloads/
  results/
  scripts/

Status: public scaffold implemented. CLI/interpreter harness implemented. Package-throughput harness implemented. Server workload smoke implemented. Server loopback runner implemented. JVM JMH smoke implemented. Historical result schema: implemented. History conversion tooling implemented. Named-machine baseline manifests implemented. Comparison reports implemented. Allocation and GC metric capture implemented for JMH. Server GC-log evidence implemented for loopback runs. Server loopback allocation evidence implemented with opt-in JFR allocation events. Server lifecycle static fixture validation implemented. History chart/report generation implemented for accepted baseline manifests. Advisory performance gates implemented. Public reference data and smoke numbers: not performance claims.

Meris keeps benchmark infrastructure in the public repository so performance work can be reviewed with the same discipline as syntax, type rules, and runtime behavior. The first scaffold is deliberately scenario-shaped: it starts from workloads that resemble code users will write, then allows narrower measurements when they explain one of those paths.

Workload Families

Family Path First purpose
Server JSON API benchmarks/workloads/server-json-api Route dispatch, JSON body fixtures, and dependency lifecycle shape
Package throughput benchmarks/workloads/package-throughput Multi-package check/compile throughput and package graph cost
CLI startup benchmarks/workloads/cli-startup Cold command latency for no-op, hello-world, and tiny project inputs
Interpreter data transform benchmarks/workloads/interpreter-data-transform Script execution over structured input fixtures
JVM hot paths benchmarks/workloads/jvm-hot-paths Generated-code behavior behind products, functions, lists, and equality

Microbenchmarks are allowed only when they explain a named production-shaped workload. A tiny loop without workload context, warmup policy, repeated runs, machine metadata, and committed history is not enough evidence.

Result Records

Benchmark history records are JSON Lines. Each record names one run of one workload under one command.

Required metadata:

  • commit,
  • Meris version,
  • timestamp,
  • operating system,
  • CPU,
  • JVM,
  • exact command,
  • warmup,
  • iterations,
  • forks,
  • workload id and path,
  • trust level,
  • noise caveats.

The canonical schema is benchmarks/results/history.schema.json. history.example.jsonl and files under benchmarks/results/reference/ are schema/reference examples until real runs are collected on named machines.

Trust Levels

Every public history record carries a trustLevel so readers can separate a working harness from evidence suitable for release notes:

Trust level Meaning
smoke The benchmark path ran. This proves wiring, not speed.
local-baseline Same-machine comparison data for development review.
release-baseline Accepted named-machine data for release comparison.
published-reference Curated public reference data intended for long-lived documentation.

Only release-baseline and published-reference records should support public performance claims. smoke and local-baseline records are useful engineering signals, but they should not be quoted as language-speed evidence.

Smoke Validation

benchmarks/scripts/smoke.sh
BENCHMARKS_SMOKE_BUILD=1 benchmarks/scripts/smoke.sh
BENCHMARKS_SMOKE_JMH=1 benchmarks/scripts/smoke.sh

The smoke script validates the scaffold and, when requested, runs the current Meris CLI and JMH smoke tasks against checkable workload fixtures. It is not a performance run. The scaffold check includes the Server JSON API static lifecycle history example and dry-runs the lifecycle runner so the public JSONL shape stays checked before any release baseline exists.

CLI And Interpreter Harness

BENCHMARKS_BUILD=1 benchmarks/scripts/run-cli-interpreter.sh

This harness runs process-isolated commands for no-op script checking, hello-world interpreter execution, tiny project checking, and the interpreter data-transform workload. It emits JSON Lines records shaped for benchmarks/results/history.schema.json. Build/install time is outside the measured command.

Useful controls:

  • BENCHMARKS_WARMUP,
  • BENCHMARKS_ITERATIONS,
  • MERIS_BIN,
  • BENCHMARKS_OUTPUT.

Package Throughput

PACKAGE_BENCHMARK_BUILD=1 benchmarks/scripts/run-package-throughput.sh

The package-throughput harness runs process-isolated meris check --project against benchmarks/workloads/package-throughput/medium-workspace. The workload is a committed multi-package graph, so the record measures package discovery, source loading, parsing, package graph checks, name resolution, and type checking together.

Useful controls:

  • PACKAGE_BENCHMARK_WARMUP,
  • PACKAGE_BENCHMARK_ITERATIONS,
  • PACKAGE_BENCHMARK_WORKLOAD,
  • MERIS_BIN,
  • PACKAGE_BENCHMARK_OUTPUT.

Each measured iteration runs meris check --project with an opt-in workspace profile. The history record includes process wall-clock duration, the Meris workspace profile total, harness overhead, and phase metrics named workspace_phase.<phase>.duration_ms. PACKAGE_BENCHMARK_OUTPUT appends JSONL records instead of replacing the file, so local history can grow over time. Allocation and GC remain marked unavailable for this harness until a named collector is added. Treat local output as same-machine regression evidence, not as a universal language-speed claim.

Server Smoke

benchmarks/scripts/run-server-smoke.sh
SERVER_BENCHMARK_SMOKE_BUILD=1 benchmarks/scripts/run-server-smoke.sh

The server smoke validates the Server JSON API workload files, the weighted request mix, dependency lifecycle fixture, and schema-only result example. It also checks the route and application source when the Meris CLI is available. It does not open a network host or report throughput.

JVM JMH Smoke

./gradlew :benchmarks:jmh:jmhSmoke
./gradlew :benchmarks:jmh:jmh

The JMH module compiles a checked Meris JVM workload during trial setup, loads the generated bytecode, and measures cached calls into generated methods. The smoke task uses one warmup iteration, one measurement iteration, and one fork only to prove the benchmark path works. Full JMH output is raw data; convert it to the public history schema before appending release comparisons.

History Conversion

benchmarks/scripts/convert-benchmark-history.py \
  --input /tmp/cli-history.jsonl \
  --input-format history-jsonl \
  --validate-only

The converter validates existing CLI/interpreter history JSONL records and can convert raw JMH JSON into the public history schema when the missing run metadata is supplied. It requires commit, version, timestamp, OS, CPU, JVM, exact command, workload identity, warmup, iterations, forks, and noise caveats.

The converter writes reviewable JSONL; it does not append records to committed history automatically. That is deliberate. Benchmark history should be a public artifact a maintainer can read before it becomes evidence.

History Charts

benchmarks/scripts/generate-benchmark-history-report.py \
  --manifest benchmarks/results/reference/release-mbp-m3-pro-01/2026-05-29-manifest.json \
  --output benchmarks/reports/2026-05-29-history.md \
  --charts-dir benchmarks/reports/2026-05-29-history-charts

The history report generator reads a named-machine baseline manifest, validates the linked history files, writes one SVG trend chart per machine/workload/metric series, and writes a Markdown traceability report. Each plotted point keeps its commit, command, machine, JVM, workload, metric source, warmup, iterations, forks, and caveats visible to reviewers.

The generator accepts only status: accepted manifests by default. --include-example is for fixture smoke checks, and --include-candidate is for local review. Non-accepted output is labeled as non-claim data.

Server Loopback

SERVER_BENCHMARK_BUILD=1 benchmarks/scripts/run-server-loopback.sh

The Server JSON API loopback runner starts the current JDK host path through meris serve, replays the weighted request mix, and records throughput, p50/p95/p99 latency, startup/shutdown duration, unexpected errors, and a history JSONL sidecar when requested.

Loopback results are useful regression evidence on the same named machine. They are not internet-scale throughput claims. Set SERVER_BENCHMARK_GC_LOG=1 to write a JVM unified GC log for the server process and include GC event count, total pause time, and max pause time in the history sidecar with source jvm-gc-log.

Set SERVER_BENCHMARK_ALLOCATION_SOURCE=jfr-tlab-events to collect allocation bytes/request from JVM Flight Recorder allocation events emitted by the server process after warmup. The runner parses jdk.ObjectAllocationInNewTLAB.tlabSize and jdk.ObjectAllocationOutsideTLAB.allocationSize. This is allocation evidence, but it has profiler overhead; compare it only with matching source settings. GC logs are not allocation evidence.

Set SERVER_BENCHMARK_FIXTURE=custom-codec to run the benchmark-owned host. That path can emit observation events for request-body disconnect probes and shutdown lifecycle. SERVER_BENCHMARK_DRAIN_DEADLINE_MILLIS controls the drain deadline passed to that host. The result JSON and history sidecar record drain-start count, configured and observed drain deadline, drain policy class, cleanup-completed count, cleanup failures, and cleanup state. The default meris serve loopback path still reports those lifecycle observations as unavailable because the benchmark runner cannot observe its sink.

Server Lifecycle Static

SERVER_LIFECYCLE_BENCHMARK_HISTORY_OUTPUT=/tmp/server-lifecycle-static.jsonl \
benchmarks/scripts/run-server-lifecycle-static.sh

run-server-lifecycle-static.sh validates benchmarks/workloads/server-json-api/fixtures/dependency-lifecycle.json without opening a server socket. It loads the committed fixture, checks the declared dependency order, verifies the expected acquire/release event shape, and writes benchmarks/results/server-json-api/lifecycle-static-latest.json by default.

Set these environment variables to control the run:

  • SERVER_LIFECYCLE_BENCHMARK_WORKLOAD,
  • SERVER_LIFECYCLE_BENCHMARK_WARMUP,
  • SERVER_LIFECYCLE_BENCHMARK_ITERATIONS,
  • SERVER_LIFECYCLE_BENCHMARK_OUTPUT,
  • SERVER_LIFECYCLE_BENCHMARK_HISTORY_OUTPUT,
  • SERVER_LIFECYCLE_BENCHMARK_TRUST_LEVEL.

When SERVER_LIFECYCLE_BENCHMARK_HISTORY_OUTPUT is set, the runner emits a meris.benchmark.history.v1 JSONL sidecar. Lifecycle measurements use the server-json-api.lifecycle-static metric prefix. Duration measurements use process-wall-clock; fixture shape measurements use lifecycle-fixture. Runtime lifecycle counters are explicitly marked unavailable because the current loopback host path cannot observe source Server.Application dependency acquire/release counters.

The committed example at benchmarks/results/examples/server-lifecycle-static-history-example.jsonl is a smoke/schema fixture. It is useful for validating history tooling and local-baseline wiring. It is not live HTTP throughput evidence, not a cancellation measurement, and not a release-baseline codec or live lifecycle counter claim. Live shutdown drain and cleanup observations are recorded by the custom-codec loopback fixture described above.

Named Baselines

benchmarks/scripts/baseline-manifest.py validate \
  benchmarks/results/baselines/example-machine-manifest.json

Named-machine baseline manifests record the machine identity, source commit, run policy, workload families, commands, and noise caveats that make benchmark history reviewable. The example manifest is a validation fixture only. Real release comparisons should use accepted manifests beside public history files.

Comparison Reports

benchmarks/scripts/compare-benchmark-history.py \
  --baseline benchmarks/results/history.jsonl \
  --current /tmp/current-history.jsonl \
  --output benchmarks/reports/YYYY-MM-DD-summary.md

Comparison reports pair measurements by workload and metric name, apply advisory thresholds from benchmarks/results/thresholds.json, and write a Markdown report with deltas, percentages, commands, workload paths, and noise caveats. These reports help review; they do not turn noisy local numbers into release facts.

Allocation And GC

benchmarks/scripts/run-jmh-allocation-gc.sh

The first allocation/GC path uses JMH GCProfiler for the generated JVM hot-path suite. Converted records carry metric-source metadata such as jmh-gc-profiler, process-wall-clock, and unavailable, so memory-pressure claims do not get mixed with ordinary elapsed-time measurements.

Server loopback allocation evidence uses opt-in JFR allocation events. The source label is jfr-tlab-events. It records bytes/request for the measured loopback run after warmup and should only be compared with runs using the same JFR settings.

Server loopback GC evidence uses JVM unified GC logs. The source label is jvm-gc-log. It records GC event count and parsed pause durations. GC logs are useful memory-pressure evidence, but they are not allocation evidence.

Performance Gates

docs/engineering/performance-gates.md

Performance gates are advisory until Meris has named-machine baselines, history conversion, and workload-specific regression thresholds. A pull request that changes a hot path should still say which benchmark path was run, which path was deferred, or which benchmark issue is needed.

The policy is intentionally narrow:

Change area First evidence
CLI startup or tiny project loading CLI/interpreter harness
Multi-package workspace checking Package-throughput harness
Interpreter execution CLI/interpreter harness and matching workload
JVM bytecode or generated runtime calls JMH smoke or full JMH run
Server route dispatch Server smoke, then loopback runner when available
Server dependency lifecycle shape Server smoke, then static lifecycle runner
Standard library hot paths Matching CLI/interpreter or JMH workload
Documentation or parser-only changes Usually no benchmark evidence

Smoke runs are health checks for the benchmark machinery. They should not be quoted as speed claims in release notes, the wiki, or the book. Release-quality comparisons require committed history records with commit, version, timestamp, machine, JVM, command, warmup, iterations, forks, workload, and noise notes.

Next Work

The scaffold and first harnesses create the public contract. The next issues should add:

  • stricter regression thresholds after enough named-machine data exists.

Related

Clone this wiki locally