Demo: three-node gossip cluster

Location: contrib/demo/cluster/

Runs three ahdapa instances on loopback ports 8080, 8081, and 8082 behind a shared self-signed TLS certificate. The script verifies that CRDT state converges across all three nodes, cross-node token issuance works, and a token issued on one node is introspected successfully on another.

What it shows

CRDT convergence — a public OAuth2 client registered on node1 appears on node2 and node3 within a few gossip intervals (configured at 2 s).
Tombstone propagation — deleting the client on node3 propagates back to node1 within the same gossip window.
Any-node token issuance — a confidential OAuth2 client registered on node1 can obtain a client_credentials access token from any of the three nodes once the CRDT has converged.
Cross-node token introspection — a token issued by node1 is accepted by node2’s /introspect endpoint; all nodes share the same signing keys via gossip.
Cross-node session revocation — enabled by distributed_mode = "eventual" in each node config. Logouts written into the revoked_sessions LwwMap propagate to all peers within one gossip round.
Scope definition replication — custom scope-to-claim mappings created on any node propagate to all peers.
Sustained multi-client traffic — three confidential clients (created on different nodes) are exercised across all nodes in rotation for 35 s; per-node latency and gossip push statistics are printed at the end.

Prerequisites

openssl(1) — for generating the demo TLS PKI (present on all modern Linux systems).
python3 — for JSON parsing in convergence checks.
Ports 8080, 8081, and 8082 free. The script evicts any stale processes bound to those ports before starting.
The ahdapa binary — the script looks in $PATH first (resolved to its full absolute path), then target/release/ahdapa, then target/debug/ahdapa, and falls back to cargo build if none is found.

Running

# Non-interactive: runs all steps, prints PASS/FAIL, exits.
contrib/demo/cluster/run.sh

# Interactive: same steps, then keeps nodes running until Ctrl-C.
contrib/demo/cluster/run.sh --interactive

What the script does

Generates a P-256 CA root and a server certificate with subjectAltName=IP:127.0.0.1, shared by all three nodes. Valid for 1 day.
Starts all three nodes with fresh SQLite databases under /tmp/.
Logs in as alice on each node, then generates a random 32-byte cluster wrapping key and pushes it to all three via PUT /api/admin/keys/cluster, then re-authenticates on node1 to obtain a cross-node session cookie.
Fetches each node’s ML-KEM-768 and ECDSA P-256 gossip keys via GET /api/gossip/kem-info and seeds them into the other two via POST /api/admin/nodes/seed so that the first encrypted gossip round can proceed.
Creates a public OAuth2 client on node1 and polls node2 and node3 until it appears (convergence check).
Deletes the client on node3 and confirms the deletion propagates to node1.
Creates a confidential client on node1 (client_secret_post), waits for convergence, then requests a client_credentials token from each node and verifies HTTP 200 with access_token on each.
Introspects the node1 token via node2’s /introspect endpoint and asserts active: true.
Creates two more clients (one on node2, one on node3), waits for convergence, then runs 35 s of rotating token traffic across all three nodes and clients.
Prints per-node token latency, per-client success rates, and gossip push statistics, then prints PASS or FAIL.

Example output (abbreviated)

Generating TLS PKI (CA + server certificate)...
  CA cert:     .../tls/ca.pem
  server cert: .../tls/cert.pem
Starting node1 (:8080)...
Starting node2 (:8081)...
Starting node3 (:8082)...
Waiting for all nodes to be ready...
  node1 ready.
  node2 ready.
  node3 ready.
Synchronizing cluster wrapping key across all nodes...
  generated cluster key: dGVzdGtleXRl...  (truncated)
  node1: cluster key set.
  node2: cluster key set.
  node3: cluster key set.
  re-authenticated on node1 with shared key.
Bootstrapping gossip KEM and signing key exchange...
  node1 KEM node_id: 127.0.0.1:8080
  node2 KEM node_id: 127.0.0.1:8081
  node3 KEM node_id: 127.0.0.1:8082
  seeded node2 keys → node1
  ...
  Waiting 5 s for first gossip rounds...

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Step 1 — create an OAuth2 client on node1
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Created client: 3f8a1b2c-...

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Step 2 — wait for gossip to replicate the client to node2 and node3
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  node2: client appeared after 3s
  node3: client appeared after 3s

  ...

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Statistics — 35s window, 11 batches × 3 clients
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  Token endpoint (by node):
  Node      Requests     Success     Avg latency
  ────────  ────────  ───────  ───────────
  node1            8   8/8          6ms
  node2            8   8/8          5ms
  node3            8   8/8          5ms
  total           24  24/24

  Gossip (outbound pushes, traffic window only):
  Node      Pushes     Total bytes     Avg/push   Skips
  ────────  ──────  ───────────  ────────  ─────
  node1          3         22500        7500       8
  node2          2         15000        7500       9
  node3          2         15000        7500       9
  total          7         52500       —         26

  Generation-skip rate: 26/33 push attempts skipped (78%)
  cluster converged — majority of gossip rounds were no-ops

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  PASS — all cluster demo steps succeeded.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Interactive exploration

With --interactive, the script keeps all three nodes running after the verification steps. The browser will warn about the self-signed certificate; add contrib/demo/cluster/tls/cert.pem to your browser’s trust store or click through the warning for local testing.

node1  https://127.0.0.1:8080/ui/
node2  https://127.0.0.1:8081/ui/
node3  https://127.0.0.1:8082/ui/

Log in as alice / alice123 on any node. The admin panel reflects the same client list on all three nodes; changes made on any node appear on the others within the gossip interval.

Configuration notes

File	Description
`node1.toml`	Port 8080, gossip peers: 8081 and 8082, `distributed_mode = "eventual"`
`node2.toml`	Port 8081, gossip peers: 8080 and 8082, `distributed_mode = "eventual"`
`node3.toml`	Port 8082, gossip peers: 8080 and 8081, `distributed_mode = "eventual"`
`users.toml`	Static users shared by all nodes; `alice` is in the `admins` group

Key settings in each node config:

[gossip]
peers            = ["https://127.0.0.1:808x", "https://127.0.0.1:808y"]
interval_secs    = 2
allowed_node_ids = ["127.0.0.1:8080", "127.0.0.1:8081", "127.0.0.1:8082"]

[cluster]
distributed_mode = "eventual"

GSSAPI/Kerberos is disabled (keytab path set to /nonexistent/keytab); alice authenticates with a password from the static users file. The [tls] section is not present in the static config files; the script appends it at runtime using the paths of the generated certificate and key.

CI usage

The script is also run as a CI integration test in the cluster-demo job of .github/workflows/ci.yml. It exits 0 on pass and 1 on any assertion failure.

Ahdapa documentation