Gossip Protocol

The gossip protocol synchronises IdpCrdt state across cluster nodes. It is implemented in src/routes/gossip.rs. The CMS envelope construction lives in crates/ahdapa-cms/.

Design

The protocol uses delta-based exchange by default, falling back to full-state on first contact or after an error. After a successful round, a node sends only the CRDT entries that changed since the last successful exchange with each peer (a sparse IdpCrdt delta), rather than the full state every time. Peers exchange generation counters in each envelope to coordinate what they already have. One round-trip still brings both nodes to the same merged state.

Full-state pushes occur in the following cases:

• First contact with a peer (no prior peer_last_gen entry).
• After any connection error or non-2xx response (generation tracking is cleared).
• When a pull_wrapping_key failure occurs (generation tracking is cleared to force retry).

Full-state exchange is the safe baseline because CRDT merge is always additive: a delta merged into a full state, or a full state merged into a delta, produces the same result as two full states merged. Old nodes that do not understand the is_delta field decode it as false (full state) and merge safely. See Payload size and bandwidth for the full bandwidth model.

Protocol

Endpoint

POST /api/gossip/sync
Content-Type: application/pkcs7-mime
X-Ahdapa-Node-Id: <sender's node_id>

<DER SignedData wrapping EnvelopedData>

The handler verifies and decrypts the message, applies admission filters, merges the received CRDT, persists the result, and replies with its own state — either a delta (when request_delta_since is set in the inbound envelope) or the full CRDT — in the same CMS format.

CMS wire format

Gossip messages use a two-layer CMS structure:

OUTER: SignedData {
  eContentType = id-envelopedData
  eContent     = <inner EnvelopedData DER>
  certificates = { sender_self_signed_cert }   // carries sender's P-256 public key
  signerInfos  = { SignerInfo {
    signatureAlgorithm = id-ecPublicKey (P-256)
    signature = ECDSA-P256-Sign(sha256(eContent))
  } }
}

INNER: EnvelopedData {
  recipientInfos = SET OF OtherRecipientInfo {
    oriType  = id-ori-kem
    oriValue = KEMRecipientInfo {
      kem           = id-alg-ml-kem-768
      kemct         = <ML-KEM-768 encapsulated ciphertext>
      wrap          = id-aes256-wrap
      encryptedKey  = AES-256-KeyWrap(kek, CEK)   // 40 bytes
    }
  }                                                // one ORI per recipient
  encryptedContentInfo {
    contentEncryptionAlgorithm = id-aes256-gcm + GcmParameters(nonce)
    encryptedContent           = AES-256-GCM(CEK, nonce, CBOR) ‖ tag[16]
  }
}

Key derivation: kek = HKDF-SHA256(ml_kem_shared_secret, info="ahdapa-cms-kek", 32)

Key material

Each node has two key pairs stored in the local node_keys table (never gossiped):

A minimal self-signed X.509 certificate for the P-256 key is also stored (signing_certificate_der) and embedded in every outbound SignedData so the receiver can extract the sender’s public key for SPKI comparison without needing a CA.

Keys are generated on first start by bootstrap_node_kem_key() in src/routes/mod.rs and reused across restarts.

Sender logic (sign_and_seal)

1. check peer_last_gen[peer] == current_gen → skip (CRDT unchanged since last sync)
2. select payload:
   - if peer_last_gen[peer] exists: crdt.delta_since(peer_last_gen[peer]) → is_delta=true
   - otherwise (first contact or after error): crdt.clone() → is_delta=false
3. serialize payload IdpCrdt to CBOR (ciborium) → crdt_bytes
4. look up peer's kem_public_key_der in local CRDT by hostname match
   → skip peer if KEM key not found
5. wrap in GossipEnvelope {
       crdt: crdt_bytes,
       issued_at: now,
       is_delta,
       my_gen: current_gen,
       request_delta_since: peer_response_gen.get(peer),  // ask peer for delta response
   }
6. serialize GossipEnvelope to CBOR → plaintext
7. ahdapa_cms::sign_and_seal(plaintext, [peer_kem_spki],
                              own_signing_priv_pkcs8, own_signing_cert_der)
   a. seal(plaintext, recipients):
      i.   generate random CEK (256-bit) and nonce (96-bit)
      ii.  AES-256-GCM(CEK, nonce, plaintext) → ciphertext ‖ tag
      iii. for each recipient: ML-KEM-768 encapsulate → (kemct, ss)
           kek = HKDF-SHA256(ss, "ahdapa-cms-kek", 32)
           AES-256 key-wrap(kek, CEK) → encryptedKey
           encode KEMRecipientInfo + OtherRecipientInfo
      iv.  EnvelopedDataBuilder.build() → enveloped_der
   b. CmsContentInfo::sign(enveloped_der, own_cert, own_priv_key) → signed_der
8. POST /api/gossip/sync with Content-Type: application/pkcs7-mime

Receiver logic (verify_and_open)

1. read X-Ahdapa-Node-Id header → sender_node_id
2. look up sender's gossip_signing_pub_key_der in local CRDT
   → None (no pinned key) → reject 401; TOFU is no longer accepted
3. ahdapa_cms::verify_and_open(body, own_kem_priv_pkcs8, sender_signing_pub_spki)
   a. cms.certs()[0] → embedded signer cert → extract SPKI
   b. compare embedded SPKI against pinned sender_signing_pub_spki; mismatch → 401
   c. cms.verify(NO_SIGNER_CERT_VERIFY) → validates ECDSA signature; → 401 on failure
   d. open(enveloped_der, own_kem_priv_pkcs8):
      i.   find OtherRecipientInfo with oriType = id-ori-kem
      ii.  ML-KEM-768 decapsulate(own_priv, kemct) → ss
           kek = HKDF-SHA256(ss, "ahdapa-cms-kek", 32)
           AES-256 key-unwrap(kek, encryptedKey) → CEK
      iii. parse GcmParameters → nonce
      iv.  AES-256-GCM decrypt(CEK, nonce, encryptedContent) → CBOR bytes
4. ciborium::from_reader(CBOR bytes) → GossipEnvelope { crdt: Vec<u8>, issued_at: i64, is_delta: bool, my_gen: u64, request_delta_since: Option<u64> }
5. reject if issued_at < now - tombstone_ttl_secs (default 7 days) → replay prevention
6. ciborium::from_reader(envelope.crdt) → peer_crdt
7. apply admission filters (see below)
8. merge + persist
9. determine response payload:
   - if envelope.request_delta_since is Some(since): crdt.delta_range(since, pre_merge_gen) → is_delta=true
   - otherwise: crdt.clone() → is_delta=false
   wrap in GossipEnvelope { crdt: CBOR(response_crdt), issued_at: now, is_delta, my_gen: post_merge_gen, request_delta_since: None }
   sign_and_seal → response

Admission filters

The receiver applies two layered filters before merging:

Layer 1 — node allowlist (gossip.allowed_node_ids): the combined static and topology-derived allowlist is always enforced. An empty union of both lists admits nobody (fail-closed). Any new NodeEntry whose node_id is not in the combined allowlist is dropped from peer_crdt before merge. Protects against rogue nodes self-registering and obtaining the cluster wrapping key.

Layer 2 — self-registration rule: a sender may only add its own NodeEntry via gossip. Any new entry (not already in the local CRDT) whose node_id does not match the X-Ahdapa-Node-Id header is dropped. This is defense-in-depth only since the header is forgeable.

KEM self-registration and signing-key pinning

A node whose KEM key is not yet in the CRDT cannot receive encrypted gossip — the sender skips peers with no known KEM key. POST /api/gossip/register-kem seeds both the ML-KEM-768 public key and the ECDSA P-256 gossip signing key before the first gossip exchange.

In IPA deployments, after each topology refresh, the local node calls register_self_with_peer() for every newly discovered peer that does not yet have this node’s KEM key. That function:

1. Acquires a Kerberos service ticket for HTTP@<peer_host> using the local machine credential (gss_initiator).
2. POSTs this node’s ML-KEM-768 public key and its ECDSA P-256 gossip signing public key to <peer_url>/api/gossip/register-kem with Authorization: Negotiate <AP-REQ>.
3. The peer verifies the AP-REQ, extracts the authenticated principal (HTTP/<hostname>@<REALM> via ServicePrincipal::parse), and stores both keys in the NodeEntry under <hostname> — provided that hostname matches the node_id in the request body, is in the allowlist, and (when gossip.kerberos_realm is set) the principal’s realm matches the expected realm.

The insert uses a three-case match: insert-fresh (neither key known), upsert-signing- key-only (KEM key known but signing key absent), or no-op (both keys already set). Once the signing key is pinned, gossip_sync rejects any message from that sender whose embedded ECDSA key does not match the pinned value — there is no TOFU fallback. This requires gssapi.initiator_principal to be set so that AppState::ipa.gss_initiator is Some; the mechanism is a no-op when it is absent.

Background loop

routes::gossip::run(state) is spawned from main.rs after AppState is constructed. When ipa_topology = true, a separate task (topology::run_topology_refresh) is also spawned; it populates AppState::dynamic_peers and AppState::dynamic_allowed_nodes before the first gossip round.

#![allow(unused)]
fn main() {
tokio::spawn(routes::gossip::run(state.clone()));
// When ipa_topology = true:
tokio::spawn(topology::run_topology_refresh(state.clone()));
}

The gossip loop maintains two per-peer maps:

• peer_last_gen[peer] — the local CRDT_GENERATION after the last successful sync with this peer. Used to skip pushing when nothing has changed locally, and to compute the delta payload (delta_since(peer_last_gen[peer])).
• peer_response_gen[peer] — the peer’s CRDT_GENERATION reported in their last response envelope. Sent back in the next push as request_delta_since so the peer can respond with only new entries it has written since that generation.

Both maps are cleared on any error so the next round falls back to a full-state exchange.

loop:
    sleep(interval_secs)  [or wake on gossip_notify signal]
    purge_expired_families(now)         // remove expired refresh families before push
    current_gen = CRDT_GENERATION.load()
    // effective peer list = gossip.peers ∪ dynamic_peers (from IPA topology)
    // stale entries for removed peers are pruned from both maps
    any_synced = false
    for each peer in (gossip.peers + state.dynamic_peers):
        if peer_last_gen[peer] == current_gen: skip (CRDT unchanged, log DEBUG)
        find peer's KEM key in local CRDT (hostname match on node_id)
        if no KEM key: warn and skip
        // Compute payload
        if peer_last_gen[peer] exists:
            payload = crdt.delta_since(peer_last_gen[peer])  // sparse delta
            is_delta = true
        else:
            payload = crdt.clone()                           // full state
            is_delta = false
        envelope = GossipEnvelope {
            crdt: CBOR(payload),
            issued_at: now,
            is_delta,
            my_gen: current_gen,
            request_delta_since: peer_response_gen.get(peer),
        }
        sign_and_seal(CBOR(envelope), [peer_kem_spki])
        POST {peer}/api/gossip/sync
        if response is 404 and peer was topology-discovered: log at DEBUG (peer not yet running ahdapa); continue
        if success:
            verify_and_open(response, own_kem_priv, peer_signing_pub)
            CBOR deserialize → peer_envelope (GossipEnvelope)
            CBOR deserialize peer_envelope.crdt → peer_crdt  // may be delta or full state
            merge into local CRDT; purge_expired_families(now)
            persist_to_db
            if persist fails: gossip_stats.persist_errors += 1
            peer_last_gen[peer] = CRDT_GENERATION.load()  // post-merge gen
            if peer_envelope.my_gen > 0:
                peer_response_gen[peer] = peer_envelope.my_gen
            if peer's wrapping_key_id ≠ local_wrapping_key_id:
                GET {peer}/api/gossip/wrapping-key
                    → SignedData(EnvelopedData) + X-Ahdapa-Node-Id header
                look up peer's pinned signing key; reject if absent
                verify_and_open(blob, own_kem_priv, peer_signing_pub) → raw_key
                update in-memory key pair; persist to node_keys
                if pull fails:
                    clear peer_last_gen[peer]; clear peer_response_gen[peer]
                    gossip_stats.wrapping_key_pull_errors += 1
            any_synced = true
        if error: clear peer_last_gen[peer]; clear peer_response_gen[peer]
    // Update round statistics — only when at least one peer synced successfully
    if any_synced:
        gossip_stats.rounds_completed += 1
        gossip_stats.last_round_at = now
    every ~1 hour:
        cleanup_expired_families(db, now)   // DB-level purge
        cleanup_old_tombstones(db, now - tombstone_ttl_secs)

If a peer is unreachable, the error is logged and the loop continues to the next peer. The loop does not back off — it retries on every interval.

The topology refresh task runs an initial fetch immediately on startup (before the first gossip sleep) and then sleeps for ipa_topology_interval_secs (minimum 30 s, default 300 s). On LDAP error, the previous peer list is kept unchanged and a warning is logged.

After each successful topology fetch, if gss_initiator is available, the topology task also calls register_self_with_peer() for each newly-discovered peer whose KEM key is not yet in the CRDT. This pre-seeds the key via POST /api/gossip/register-kem with a Kerberos AP-REQ so that the legitimate node wins the OR-Map first-write-wins race before the first gossip round fires.

Cluster wrapping key

The 32-byte cluster wrapping key (used for session cookies) is stored node-locally in node_keys.wrapping_key_cms_der as a CMS EnvelopedData blob sealed to the node’s own ML-KEM-768 public key. It is never gossiped in plaintext or as a multi-recipient blob.

Only a short UUID string (wrapping_key_id) is gossiped in the CRDT. When a node observes a different UUID after a gossip merge, it fetches the actual key on demand:

GET /api/gossip/wrapping-key
X-Ahdapa-Node-Id: <requester's node_id>

Response: 200 OK
Content-Type: application/octet-stream
X-Ahdapa-Node-Id: <responder's node_id>
Body: SignedData(EnvelopedData) DER

The response is a full SignedData(EnvelopedData) blob produced by sign_and_seal(), sealed to exactly one recipient (the requester’s ML-KEM-768 public key) and signed with the responder’s ECDSA P-256 gossip signing key. The requester looks up the responder’s pinned signing key in the CRDT (from the X-Ahdapa-Node-Id header) and calls verify_and_open(). A response from a node with no pinned signing key is rejected. Confidentiality is ensured by the inner ML-KEM-768 encryption; integrity and sender authentication are ensured by the outer ECDSA P-256 signature.

Node statistics endpoint

GET /api/gossip/stats

Unauthenticated. Intentionally unauthenticated — like /api/gossip/kem-info — because the admin web UI fetches it before an admin session is established. What is exposed is aggregate counts and gossip health indicators; no key material, user data, or token content is returned.

Response body (JSON):

{
  "node_id":          "ipa1.example.com",
  "crdt_generation":  42,
  "counts": {
    "clients":          3,
    "signing_keys":     2,
    "cluster_nodes":    3,
    "refresh_families": 7,
    "revoked_sessions": 1,
    "scope_definitions": 8,
    "ipa_idp_overrides": 0
  },
  "peers":               ["https://ipa2.example.com/idp", "https://ipa3.example.com/idp"],
  "active_signing_kid":  "abc123",
  "kem_enrolled":        true,
  "gossip_signing_enrolled": true,
  "gossip": {
    "started_at":      1716000000,
    "rounds_completed": 12,
    "last_round_at":   1716000060,
    "peer_last_sync":  { "ipa2.example.com": 1716000058, "ipa3.example.com": 1716000059 },
    "persist_errors":  0,
    "wrapping_key_pull_errors": 0
  }
}

Field notes:

• crdt_generation — current value of the CRDT_GENERATION atomic counter.
• counts.* — live (non-tombstoned) entry counts for each CRDT collection.
• peers — union of configured gossip.peers and topology-discovered peers.
• active_signing_kid — the kid of the currently active JWT signing key.
• kem_enrolled / gossip_signing_enrolled — whether both cryptographic identities are registered in the CRDT.
• gossip.started_at — Unix timestamp when the gossip background task started.
• gossip.rounds_completed — number of gossip rounds in which at least one peer was successfully synced. Idle rounds (CRDT unchanged, all pushes skipped) and rounds where all peers fail do not increment this counter.
• gossip.last_round_at — Unix timestamp of the most recent round that synced at least one peer. null until the first successful sync.
• gossip.peer_last_sync — Unix timestamp of the most recent successful inbound sync from each peer (recorded by the /api/gossip/sync receiver).
• gossip.persist_errors — cumulative DB persist failures since startup (incremented after both inbound sync and outbound merge failures).
• gossip.wrapping_key_pull_errors — cumulative failures to pull the cluster wrapping key from a peer after detecting a UUID change.

This endpoint is used by the admin web UI Cluster Nodes page to display per-node runtime gossip health alongside the static CRDT node entries from GET /api/admin/nodes.

Kerberos KEM self-registration endpoint

POST /api/gossip/register-kem
Authorization: Negotiate <base64-AP-REQ>
Content-Type: application/json

{
  "node_id":                  "<hostname>",
  "kem_public_key_der":       "<base64url-ML-KEM-768-SPKI-DER>",
  "gossip_signing_pub_key_der": "<base64url-ECDSA-P256-SPKI-DER>"
}

Used by topology-discovered peers to seed both their ML-KEM-768 public key and their ECDSA P-256 gossip signing key before the first gossip round. All three fields are required; missing or empty fields return 400 Bad Request. The server:

1. Returns 503 Service Unavailable if the GSSAPI server credential is unavailable (state.gss_cred is None — indicates a configuration or keytab problem).
2. Calls try_spnego() to accept the Kerberos AP-REQ. Returns 401 Negotiate if absent, 401 if the token is invalid.
3. Calls ServicePrincipal::parse() on the authenticated principal. Rejects with 403 if the principal is not HTTP/<host>@<REALM> (user principals and non-HTTP service types are excluded).
4. When gossip.kerberos_realm is set, rejects with 403 if the principal’s realm does not match — prevents cross-realm trust escalation.
5. Checks that req.node_id.to_lowercase() == authed_host. Rejects with 403 if they differ — a machine can only register its own identity.
6. Checks that authed_host is in the topology-derived or static allowlist. Rejects with 403 if not admitted.
7. Applies a three-case match on the existing CRDT entry for this node_id:
   • Insert-fresh: neither key known → insert NodeEntry with both keys.
   • Upsert-signing-key-only: KEM key present but gossip_signing_pub_key_der empty → update the entry to add the signing key.
   • No-op: both keys already present → return 200 OK immediately (idempotent).
8. Returns 200 OK, optionally with a WWW-Authenticate: Negotiate <mutual-auth-token> header if GSSAPI produced a mutual-authentication output token.

At startup, bootstrap_wrapping_key() reads node_keys.wrapping_key_cms_der. If present, it decrypts the blob to recover the 32-byte key. If absent (first start), it generates a fresh key, seals it to the node’s own KEM key, and stores the result in node_keys. A UUID is generated and published to the CRDT as wrapping_key_id with timestamp=1 so that the established cluster’s UUID wins the LWW merge on the first gossip round.

When the cluster wrapping key is rotated via PUT /api/admin/keys/cluster, the node re-seals the new key to its own KEM key, stores it in node_keys, and updates crdt.wrapping_key_id to a new UUID. Peers detect the UUID change via gossip and pull the new key via the on-demand endpoint.

Convergence

New signing key propagation: a key added on node A is available on node B after at most 1 gossip round from A to B. Resource servers should cache JWKS with a short TTL (≤ interval_secs × 2) to avoid key-not-found errors during propagation.

Security considerations

• /api/gossip/sync SHOULD be firewalled to the cluster’s subnet as defense-in-depth. CMS encryption ensures confidentiality even if traffic is captured, but network isolation prevents unauthorized nodes from attempting to self-register.
• The allowlist is fail-closed. When both the static allowed_node_ids list and the topology-derived allowlist are empty, no node can self-register via gossip or the wrapping-key endpoint. This is intentional: operators must either configure an explicit allowlist or enable ipa_topology so that hostnames are discovered automatically.
• Gossip envelopes carry a timestamp (issued_at). Envelopes older than tombstone_ttl_secs (default 7 days) are rejected, preventing an attacker from replaying a captured gossip message after its tombstones have been GC-purged.
• The gossip reqwest::Client has a 10-second request timeout. Slow peers do not block the gossip loop.
• ECDSA P-256, not Ed25519, is used for gossip signing. OpenSSL’s CMS_sign() API requires a key type that has a default digest algorithm; Ed25519 (PureEdDSA) does not satisfy this requirement. P-256 provides the same 128-bit security level.
• The JWT signing algorithm is configurable, not gossip signing. Each node generates its own JWT signing key pair (algorithm set by [server] jwt_signing_algorithm, default: ES256) stored in node_keys.jwt_signing_priv_der. The private key never leaves the node; only the public key is gossiped in SigningKeyEntry. This is distinct from the ECDSA P-256 gossip signing key.

Payload size and bandwidth

Raw field sizes

The binary fields that dominate gossip payload size (measured from a three-node demo cluster):

The ML-KEM-768 public key is the dominant field by a factor of ~13× over the next largest gossiped value.

Per-entity CBOR contribution

The CRDT is serialised as CBOR (ciborium). CBOR stores binary fields as raw bytes (no base64 overhead). Approximate CBOR size per entry:

CMS envelope overhead

Each gossip message is sent to exactly one peer, so the CMS overhead is constant regardless of cluster size:

Wire size per gossip push

Total CMS-encrypted wire size for one outbound push (one recipient). The cluster wrapping key blob is no longer in the gossip body; only its UUID is gossiped:

Bandwidth per gossip cycle

The topology is full-mesh: each node pushes to every configured peer and receives a response. Total cluster bandwidth per cycle (worst case, full-state) = N × (N−1) × 2 × wire_bytes. In practice, delta exchange reduces per-push payload to the size of changed entries only.

These figures are theoretical maximums — they assume every gossip round produces an actual push. In practice, the generation-skip optimisation suppresses pushes when the CRDT has not changed since the last successful round. In the demo cluster (active token issuance, no schema mutations), 93% of rounds were skipped, reducing steady-state bandwidth to near zero. Pushes happen only when the CRDT actually changes (client creation/deletion, key rotation, node join/leave).

At interval_secs = 2 (demo default; config default is 5 s):

The O(N²) topology is practical for the expected deployment range of 2–5 nodes. Above ~10 nodes the bandwidth cost becomes significant and a partial-mesh peer configuration (each node lists only a subset of peers) should be considered.

Marginal cost per added entity

Measured at a three-node baseline:

Sessions and clients are cheap. Nodes are the dominant cost because every node contributes 1,206 B of ML-KEM-768 public key material (gossiped in NodeEntry). The previous per-node cost of +1,644 B from the cluster wrapping key CMS blob is eliminated — the wrapping key is no longer gossiped.

Tombstone accumulation and GC

Removed nodes, deleted clients, and revoked signing keys leave OrMap tombstones. Each tombstone adds ~220 B to gossip messages until it is garbage-collected. Tombstones older than gossip.tombstone_ttl_secs (default: 7 days) are purged from both the in-memory CRDT and the database approximately once per hour. This bounds tombstone growth even in high-churn deployments.

The TTL must exceed the longest expected node downtime: a node that is offline longer than the TTL may re-gossip entries that were since deleted (those entries would be re-merged on reconnection). The default 7-day TTL is conservative and suitable for most deployments.

Configuration

[gossip]
peers              = ["https://node2.example.com:8080", "https://node3.example.com:8080"]
interval_secs      = 5
allowed_node_ids   = ["node1.example.com", "node2.example.com", "node3.example.com"]
tombstone_ttl_secs = 604800  # 7 days

For IPA-integrated deployments, the static peers list can be omitted entirely when ipa_topology = true. A single-node deployment requires no gossip configuration at all.