Cross-BAC Benchmarking

You need to compare HBAC and ABAC performance on identical rule sets. The challenge: HBAC fixtures use a three-dimensional user/host/service model, while ABAC uses arbitrary dimensions. How can you benchmark both implementations fairly?

The bac-perf tool solves this through automatic conversion: ABAC can load HBAC fixtures by mapping the three dimensions to equivalent ABAC dimensions. This enables direct performance comparison without maintaining separate fixture sets.

The mapping

HBAC’s fixed dimensions map 1:1 to ABAC dimensions:

Groups are prefixed with group: in ABAC to distinguish them from primary values. The conversion preserves:

• Semantic equivalence – same allow/deny decisions
• Rule types – Allow vs Deny
• Enabled status – active rules only
• Group membership – all groups converted

Basic cross-benchmark

Generate one HBAC fixture, benchmark it with both implementations:

# 1. Generate HBAC fixture
bac-perf generate --bac-type hbac --count 1000 --output shared_1k --seed 42

# 2. Benchmark with HBAC
bac-perf bench --bac-type hbac --fixture shared_1k --scenario throughput \
  --format json --output hbac_results.json

# 3. Benchmark with ABAC (automatic conversion)
bac-perf bench --bac-type abac --fixture shared_1k --scenario throughput \
  --format json --output abac_results.json

# 4. Compare
bac-perf compare hbac_results.json abac_results.json

The fixture is loaded once, converted automatically when used with ABAC, and benchmarked under identical conditions.

Complete comparison workflow

Compare all scenarios at multiple scales:

# Generate fixtures
for size in 100 1000 10000; do
  bac-perf generate --bac-type hbac --count $size --output shared_${size} --seed 42
done

# Benchmark HBAC
for fixture in shared_100 shared_1000 shared_10000; do
  bac-perf bench --bac-type hbac --fixture $fixture --scenario all \
    --format json --output ${fixture}_hbac.json
done

# Benchmark ABAC (same fixtures)
for fixture in shared_100 shared_1000 shared_10000; do
  bac-perf bench --bac-type abac --fixture $fixture --scenario all \
    --format json --output ${fixture}_abac.json
done

# Compare each scale
for size in 100 1000 10000; do
  echo "=== Comparison at ${size} rules ==="
  bac-perf compare shared_${size}_hbac.json shared_${size}_abac.json
done

Example results

Baseline (Before ABAC Optimization)

Using a shared_1000 fixture with 1,000 rules, showing the initial implementation gap:

HBAC (native):

Scenario: throughput
Results:
  Throughput:       399K req/s
  Mean latency:     2.50 µs
  P95 latency:      3.50 µs
  P99 latency:      3.92 µs

ABAC (baseline, unoptimized):

Scenario: throughput
Results:
  Throughput:       4.2K req/s
  Mean latency:     237 µs
  P95 latency:      417 µs
  P99 latency:      552 µs

Historical gap: HBAC was ~95× faster (baseline)

Current (After Bitmap Deny Index + Arc<str> Migration)

After implementing bitmap deny index in both engines, Arc<str> migration in HBAC, and compiled evaluator in HBAC, there is no crossover – HBAC is faster or equal at all scales on x86_64:

At 1,000 rules (x86_64, mixed workload):

HBAC:   5.24M req/s  (191 ns mean)
ABAC:   4.02M req/s  (249 ns mean)
Ratio:  HBAC is 1.3× faster

At 100,000 rules (x86_64, mixed workload):

HBAC:   1.54M req/s  (651 ns mean)
ABAC:   1.22M req/s  (819 ns mean)
Ratio:  HBAC is 1.3× faster

At 1,000,000 rules (x86_64, mixed workload):

HBAC:   201K req/s   (4.98 µs mean)
ABAC:   192K req/s   (5.21 µs mean)
Ratio:  ~equal

Key insight: With bitmap deny index optimization ported to HBAC and memory-optimized rule storage (index-based references, Arc<str> categories), HBAC is faster or equal at all scales. ABAC’s advantage is N-dimensional flexibility and 3.2x less memory (76 MB vs 241 MB at 1M rules).

Understanding the performance characteristics

Both engines now share bitmap deny index optimization. HBAC is faster at all scales due to its specialized three-dimensional structure:

At all scales: HBAC’s specialization (direct field access, pre-compiled decision tree, Arc<str> deduplication) provides a consistent advantage.

ABAC’s advantages are N-dimensional flexibility and 3.2x less memory per rule, not throughput.

Use cases

1. Migration planning

Measure the performance impact before migrating from HBAC to ABAC:

# Current HBAC performance
bac-perf bench --bac-type hbac --fixture prod_rules

# Expected ABAC performance
bac-perf bench --bac-type abac --fixture prod_rules

If the performance difference is acceptable, ABAC’s flexibility may justify the overhead. If not, stay with HBAC or optimize hot paths.

2. Regression detection

Validate that ABAC optimizations don’t break correctness:

# Baseline HBAC (known correct)
bac-perf bench --bac-type hbac --fixture test_5k --format json --output baseline.json

# ABAC after optimization changes
bac-perf bench --bac-type abac --fixture test_5k --format json --output current.json

# Compare -- decisions should be identical, latency may differ
bac-perf compare baseline.json current.json

3. Trade-off analysis

Quantify the flexibility vs performance trade-off:

# Generate at multiple scales
for size in 100 1000 10000; do
  bac-perf generate --bac-type hbac --count $size --output scale_${size} --seed 42
  
  # Benchmark both
  bac-perf bench --bac-type hbac --fixture scale_${size} --scenario throughput -o hbac_${size}.json
  bac-perf bench --bac-type abac --fixture scale_${size} --scenario throughput -o abac_${size}.json
done

# Extract throughput ratios
for size in 100 1000 10000; do
  hbac_tput=$(jq '.requests_per_second' hbac_${size}.json)
  abac_tput=$(jq '.requests_per_second' abac_${size}.json)
  ratio=$(echo "scale=2; $hbac_tput / $abac_tput" | bc)
  echo "${size} rules: HBAC is ${ratio}× faster than ABAC"
done

Conversion guarantees

The HBAC → ABAC converter provides:

✓ Semantic equivalence
Same allow/deny decisions for all requests.

✓ Rule preservation
Names, types (Allow/Deny), and enabled status preserved.

✓ Group membership
All user/host/service groups mapped to ABAC groups with group: prefix.

× NOT reversible
Native ABAC fixtures (4+ dimensions, custom types) cannot be converted to HBAC.

Limitations

What you CAN do

• Run HBAC fixtures with both HBAC and ABAC
• Compare performance on identical rule sets
• Validate ABAC correctness against HBAC baseline
• Measure overhead of generic vs specialized engines

What you CANNOT do

• Run native ABAC fixtures with HBAC (no reverse conversion)
• Use ABAC-specific features (4+ dimensions, IpCidr types, custom matchers) with HBAC fixtures
• Expect identical performance (ABAC is generic, HBAC is specialized)

Best practices

1. Same seed – Always use --seed 42 for reproducible fixtures.
2. Same scenarios – Run the same benchmark scenarios for both implementations.
3. Multiple runs – Repeat benchmarks 3-5 times and average results to reduce variance.
4. Release mode – Always benchmark in --release mode.
5. Cache consistency – Use --cache for both or neither (not one).

What to read next

• CLI Reference – full option reference for bac-perf.
• Running Benchmarks – interpreting benchmark results.
• Performance Results – HBAC performance data across all scales.