Running Benchmarks

The bac-perf bench command measures evaluation latency, throughput, policy build time, and memory usage against a generated fixture. This page covers the available scenarios, how to interpret results, and how to detect regressions.

Basic benchmarking

# Run all scenarios with caching enabled (HBAC)
bac-perf bench --bac-type hbac --fixture hbac_1k_prod --scenario all --cache

# Run a specific scenario
bac-perf bench --bac-type hbac --fixture hbac_1k_prod --scenario single-latency --cache

# Save results as JSON for later comparison
bac-perf bench \
  --bac-type hbac \
  --fixture hbac_1k_prod \
  --scenario all \
  --cache \
  --format json \
  --output results/hbac_baseline.json

# Run ABAC benchmarks
bac-perf bench --bac-type abac --fixture abac_1k_prod --scenario all --cache

Scenarios

single-latency

Measures individual request evaluation time with a warm cache.

• 10,000 matching requests with 1,000 warmup iterations.
• Reports latency percentiles (min, mean, median, P95, P99, max).
• Represents best-case performance: cache is warm, all requests match.

bac-perf bench --bac-type hbac --fixture hbac_1k_prod --scenario single-latency --cache

uncached-latency

Measures raw evaluation performance without cache benefit.

• 10,000 unique non-matching requests, no warmup.
• Cache is enabled but never hit (every request is unique).
• Isolates Bloom filter, decision tree, and sequential evaluation cost.

The latency distribution is typically bimodal: most requests are rejected quickly by the Bloom filter (sub-microsecond), while the few that pass through trigger O(n) sequential evaluation and produce high-latency outliers. This is why P95 can be lower than the mean.

bac-perf bench --fixture hbac_1k_prod --scenario uncached-latency

throughput

Sustained load with a mixed request profile.

• 10,000 requests (80% matching, 20% non-matching) with 1,000 warmup.
• Reports requests per second under realistic cache pressure.

bac-perf bench --fixture hbac_1k_prod --scenario throughput --cache

build-time

Policy initialization performance.

• Measures rule loading, deserialization, and index construction time.
• Reports memory allocation.

bac-perf bench --fixture hbac_1k_prod --scenario build-time

Comparing runs

Save results as JSON and use bac-perf compare to detect regressions:

# Baseline (before change)
bac-perf bench --fixture hbac_1k --scenario all --cache \
  --format json --output baseline.json

# Current (after change)
bac-perf bench --fixture hbac_1k --scenario all --cache \
  --format json --output current.json

# Compare
bac-perf compare baseline.json current.json

The comparison shows percentage changes for each metric with pass/fail indicators.

JIT comparison

Build with JIT support and compare:

cargo build --release --features jit -p perf-testing

# Without JIT
bac-perf bench --bac-type hbac --fixture hbac_10k --scenario uncached-latency \
  --format json --output no_jit.json

# With JIT
bac-perf bench --bac-type hbac --fixture hbac_10k --scenario uncached-latency --jit \
  --format json --output with_jit.json

bac-perf compare no_jit.json with_jit.json

JIT compilation provides improvement for uncached evaluation. It has no effect on cached lookups. See Performance Results for measured JIT impact.

Criterion integration

For statistical rigor, use the Criterion benchmarks:

cd crates/perf-testing
cargo bench

# View HTML reports
open target/criterion/report/index.html

Criterion provides outlier detection, variance analysis, historical comparison, and regression detection with statistical confidence intervals.

Interpreting results

Cached evaluation

With caching enabled, expect sub-microsecond mean latency across all rule counts. The optimization stack (Bloom filter, decision tree, indexed cache, deny-rule index) handles the heavy lifting.

Uncached evaluation

Without cache benefit, latency scales roughly linearly with rule count. The Bloom filter rejects most requests quickly, but false positives trigger sequential evaluation at O(n) cost.

Warning signs

• P99 much higher than mean (cached): investigate cache misses or system noise.
• P95 lower than mean (uncached): normal bimodal distribution from Bloom filter rejection.
• Cache hit rate below 90%: consider pre-warming or reviewing request diversity.
• Build time above 1 second: expected at 100K+ rules. Consider lazy loading for hot-reload scenarios.

See Performance Results for benchmark data across all rule counts.

Regression detection workflow

# 1. Generate a standard fixture (same seed for consistency)
bac-perf generate --count 1000 --distribution sssd-prod --output regression_1k --seed 42

# 2. Run on main branch
git checkout main
cargo build --release -p perf-testing
bac-perf bench --fixture regression_1k --scenario all --cache \
  --format json --output before.json

# 3. Run on feature branch
git checkout feature-branch
cargo build --release -p perf-testing
bac-perf bench --fixture regression_1k --scenario all --cache \
  --format json --output after.json

# 4. Compare
bac-perf compare before.json after.json

Scaling analysis

Test across multiple rule counts to understand how your workload scales:

for size in 100 1000 10000 100000; do
  bac-perf bench --fixture hbac_${size}_baseline --scenario single-latency \
    --cache --format json --output cached_${size}.json
  bac-perf bench --fixture hbac_${size}_baseline --scenario uncached-latency \
    --format json --output uncached_${size}.json
done

Output formats

Next steps

For the complete list of command options, flags, and environment variables, see the CLI reference.

See CLI Reference.