cortivex-consensus

Cortivex Consensus Protocol

This skill covers how ConsensusManager nodes enable Raft-style leader election in multi-node Cortivex clusters. Leader election ensures that exactly one coordinator is making scheduling decisions at any time, preventing conflicting task assignments and split-brain scenarios.

Overview

When running Cortivex pipelines across multiple nodes (separate machines or separate processes), the nodes must agree on which one is the leader. The ConsensusManager node implements a simplified Raft protocol to achieve this:

• Exactly one node is elected leader and coordinates all task scheduling
• All other nodes are followers that accept work from the leader
• If the leader fails, a new election occurs automatically
• A term number tracks each leadership epoch to prevent stale commands

When to Use

• Running Cortivex agents on more than one machine or process
• Pipelines where multiple SwarmCoordinator instances must agree on a single source of truth
• Fault-tolerant deployments where leader failure should not halt the pipeline
• Environments where you need to scale horizontally by adding more Cortivex nodes
• Any scenario where two agents could both attempt to act as coordinator simultaneously

You do NOT need consensus for:

• Single-node pipelines (the lone node is automatically the leader)
• Simple DAG pipelines without SwarmCoordinator nodes
• Pipelines with a fixed, non-redundant coordinator

How It Works

Node Roles

| Role | Responsibilities | |------|-----------------| | Leader | Schedules tasks, sends heartbeats to followers, replicates state | | Follower | Accepts task assignments from leader, votes in elections | | Candidate | Temporary state during an election; requests votes from peers |

Election Flow

1. A follower has not received a heartbeat from the leader within the
   election timeout (default: 2000ms, randomized to prevent ties)
2. The follower becomes a Candidate and increments the term to T+1
3. The Candidate votes for itself and sends RequestVote RPCs to all peers
4. If the Candidate receives votes from a majority of nodes, it becomes
   Leader at term T+1
5. The new Leader sends an immediate heartbeat to all followers to
   establish authority and suppress further elections
6. Followers that receive the heartbeat at term T+1 accept the new leader

Quorum Requirements

| Cluster Size | Majority (Quorum) | Tolerated Failures | |--------------|--------------------|--------------------| | 1 | 1 | 0 | | 3 | 2 | 1 | | 5 | 3 | 2 | | 7 | 4 | 3 |

A cluster of 3 nodes is the minimum for meaningful fault tolerance. Use odd numbers to avoid tie elections.

Pipeline Configuration

Single-Node (Development)

In a single-node setup, no explicit ConsensusManager is needed. The SwarmCoordinator automatically assumes leadership:

name: single-node-pipeline
version: "1.0"
description: Single coordinator, no consensus needed
nodes:
  - id: coordinator
    type: SwarmCoordinator
    config:
      pool_size: 3
      runtime: auto

Multi-Node Cluster

For multi-node deployments, add a ConsensusManager node to each pipeline instance and configure peers:

name: clustered-pipeline
version: "1.0"
description: Three-node cluster with Raft consensus
orchestration:
  mode: cluster
  node_id: node-1
  peers:
    - id: node-2
      address: 192.168.1.12:9100
    - id: node-3
      address: 192.168.1.13:9100

nodes:
  - id: consensus
    type: ConsensusManager
    config:
      election_timeout_ms: 2000
      heartbeat_interval_ms: 500
      max_election_rounds: 10
      quorum_size: auto

  - id: coordinator
    type: SwarmCoordinator
    depends_on: [consensus]
    config:
      pool_size: 2
      runtime: auto
      leader_only: true

  - id: monitor
    type: AgentMonitor
    depends_on: [coordinator]
    config:
      auto_recovery: true

Start each node with its identity:

# Node 1
cortivex serve --port 9100 --node-id node-1 \
  --peers "node-2@192.168.1.12:9100,node-3@192.168.1.13:9100"

# Node 2
cortivex serve --port 9100 --node-id node-2 \
  --peers "node-1@192.168.1.11:9100,node-3@192.168.1.13:9100"

# Node 3
cortivex serve --port 9100 --node-id node-3 \
  --peers "node-1@192.168.1.11:9100,node-2@192.168.1.12:9100"

Verifying the Cluster

After all nodes start, verify that a leader has been elected and all nodes agree:

cortivex_cluster({
  action: "status"
})

Expected output:

{
  "cluster_size": 3,
  "leader": "node-1",
  "term": 1,
  "status": "healthy",
  "nodes": [
    { "id": "node-1", "role": "leader",   "term": 1, "last_heartbeat": "2s ago" },
    { "id": "node-2", "role": "follower", "term": 1, "last_heartbeat": "1s ago" },
    { "id": "node-3", "role": "follower", "term": 1, "last_heartbeat": "1s ago" }
  ]
}

All nodes must report the same leader and term. If they disagree, see Troubleshooting below.

Handling Node Failures

Leader Failure

When the leader node goes down:

1. Followers stop receiving heartbeats
2. After the election timeout, the first follower to time out becomes a Candidate
3. The Candidate requests votes and, with a majority, becomes the new Leader
4. The new Leader takes over task scheduling
5. In-progress tasks on the dead leader's agents are requeued

Recovery time is typically under 5 seconds.

Follower Failure

When a follower node goes down:

1. The Leader stops receiving acknowledgments from that follower
2. The Leader continues operating as long as a quorum remains
3. Tasks assigned to agents on the dead follower are requeued
4. When the follower recovers, it rejoins and catches up

Network Partition (Split Brain)

If the network splits the cluster into two groups:

• The group with a majority continues operating normally and can elect a leader
• The group with a minority cannot form a quorum and halts scheduling
• When the partition heals, minority nodes rejoin and accept the majority's state

This prevents conflicting task assignments across partitions.

ConsensusManager Node Reference

- id: consensus
  type: ConsensusManager
  config:
    election_timeout_ms: 2000           # base timeout before starting election
    election_timeout_jitter_ms: 1000    # randomized jitter added to timeout
    heartbeat_interval_ms: 500          # leader heartbeat frequency
    max_election_rounds: 10             # give up after this many failed elections
    quorum_size: auto                   # auto = (cluster_size / 2) + 1
    persist_state: true                 # persist term and vote to disk
    state_directory: .cortivex/raft/    # directory for persistent state
    log_elections: true                 # log election events for debugging

Troubleshooting

No Leader Elected

Symptoms: All nodes report role: candidate or role: follower with no leader.

Causes and fixes:

• Insufficient quorum -- Ensure a majority of nodes are running and reachable. A 3-node cluster needs at least 2 nodes.
• Incorrect peer addresses -- Verify the --peers flag on each node points to the correct addresses and ports.
• Firewall blocking -- Ensure the ports used for peer communication are open between all nodes.
• Simultaneous startup -- Stagger node startup by 1-2 seconds to reduce election ties.

Frequent Re-Elections

Symptoms: The term number increments rapidly; different nodes keep becoming leader.

Causes and fixes:

• Network instability -- Packet loss causes missed heartbeats. Increase heartbeat_interval_ms and election_timeout_ms.
• Overloaded nodes -- High CPU usage delays heartbeat processing. Reduce agent count or add resources.
• Clock skew -- Large time differences between nodes can cause timeout miscalculations. Synchronize clocks with NTP.

Split Brain Detected

Symptoms: Two nodes both claim to be leader at different terms.

Resolution: This is expected during a transient partition. The node w