> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/temporalio/temporal/llms.txt
> Use this file to discover all available pages before exploring further.

# Multi-Node Clustering

> Set up and configure multi-node Temporal Server clusters for high availability

This guide covers deploying Temporal Server in a multi-node cluster configuration for high availability, fault tolerance, and horizontal scalability.

## Cluster Architecture

A production Temporal cluster typically consists of:

<Steps>
  <Step title="Multiple Service Instances">
    Each Temporal service (Frontend, History, Matching, Worker) runs multiple instances across different nodes
  </Step>

  <Step title="Membership Protocol">
    Services use the Ringpop gossip protocol for cluster membership and peer discovery
  </Step>

  <Step title="Consistent Hashing">
    History shards are distributed across History service instances using consistent hashing
  </Step>

  <Step title="Shared Database">
    All instances connect to the same persistence and visibility databases
  </Step>
</Steps>

## Service Roles and Scaling

### Frontend Service

<Card title="Stateless - Horizontal Scaling" icon="scale-balanced">
  The Frontend service is completely stateless and can be scaled horizontally.

  **Scaling Guidelines**:

  * Start with 2-3 instances for HA
  * Add instances based on request rate
  * Each instance handles \~10k requests/sec
  * Use load balancer for traffic distribution
</Card>

```yaml theme={null}
services:
  frontend:
    rpc:
      grpcPort: 7233
      membershipPort: 6933
      bindOnIP: "0.0.0.0"
      httpPort: 7243
```

### History Service

<Card title="Stateful - Shard-Based Scaling" icon="database">
  The History service manages workflow execution state using consistent hashing of history shards.

  **Scaling Guidelines**:

  * Minimum 3 instances for HA
  * Shards automatically redistribute on instance changes
  * Each instance handles multiple shards
  * More instances = better shard distribution
</Card>

```yaml theme={null}
persistence:
  numHistoryShards: 512  # More shards = better distribution

services:
  history:
    rpc:
      grpcPort: 7234
      membershipPort: 6934
      bindOnIP: "0.0.0.0"
```

**Shard Distribution Example**:

```
512 shards, 3 History instances:
- Instance 1: ~170 shards
- Instance 2: ~171 shards  
- Instance 3: ~171 shards

512 shards, 5 History instances:
- Each instance: ~102 shards
```

### Matching Service

<Card title="Semi-Stateful - Queue-Based" icon="list-check">
  The Matching service routes tasks to workers and maintains task queue state.

  **Scaling Guidelines**:

  * Start with 2-3 instances
  * Scale based on task queue throughput
  * Task queues are partitioned across instances
</Card>

```yaml theme={null}
services:
  matching:
    rpc:
      grpcPort: 7235
      membershipPort: 6935
      bindOnIP: "0.0.0.0"
```

### Worker Service

<Card title="Stateless - Background Operations" icon="gears">
  The Worker service executes internal system workflows like archival.

  **Scaling Guidelines**:

  * Usually 1-2 instances sufficient
  * Scale if system workflow backlog occurs
</Card>

```yaml theme={null}
services:
  worker:
    rpc:
      grpcPort: 7239
      membershipPort: 6939
      bindOnIP: "0.0.0.0"
```

## Cluster Membership

Temporal services use Ringpop for cluster membership and service discovery.

### Membership Configuration

```yaml theme={null}
global:
  membership:
    maxJoinDuration: 30s
    broadcastAddress: ""  # Auto-detected if empty
```

### How Membership Works

<Steps>
  <Step title="Service Startup">
    Each service instance binds to its membership port and announces itself
  </Step>

  <Step title="Peer Discovery">
    Instances discover each other through the shared database (cluster\_membership table)
  </Step>

  <Step title="Gossip Protocol">
    Ringpop gossip maintains consistent view of cluster membership
  </Step>

  <Step title="Health Checking">
    Services continuously monitor peer health and detect failures
  </Step>

  <Step title="Shard Rebalancing">
    When topology changes, History shards automatically redistribute
  </Step>
</Steps>

### Broadcast Address

The broadcast address is how services advertise themselves to peers:

<Tabs>
  <Tab title="Auto-Detection">
    ```yaml theme={null}
    global:
      membership:
        broadcastAddress: ""  # Auto-detect from hostname
    ```

    Best for:

    * Simple deployments
    * Kubernetes with pod DNS
    * Docker with proper networking
  </Tab>

  <Tab title="Explicit IP">
    ```yaml theme={null}
    global:
      membership:
        broadcastAddress: "10.0.1.42"
    ```

    Best for:

    * Multiple network interfaces
    * NAT environments
    * Complex network topologies
  </Tab>

  <Tab title="Environment Variable">
    ```yaml theme={null}
    global:
      membership:
        broadcastAddress: "{{ env \"POD_IP\" }}"
    ```

    Best for:

    * Kubernetes deployments
    * Dynamic IP assignment
  </Tab>
</Tabs>

## Multi-Node Deployment Examples

### Docker Compose Cluster

<CodeGroup>
  ```yaml docker-compose-cluster.yml theme={null}
  services:
    # Frontend instances
    frontend-1:
      image: temporalio/server:latest
      container_name: temporal-frontend-1
      environment:
        - SERVICES=frontend
        - BIND_ON_IP=0.0.0.0
        - TEMPORAL_BROADCAST_ADDRESS=frontend-1
        - NUM_HISTORY_SHARDS=128
        - DB=postgres12
        - POSTGRES_SEEDS=postgres
        - POSTGRES_USER=temporal
        - POSTGRES_PWD=temporal
      networks:
        - temporal-network
      depends_on:
        - postgres
    
    frontend-2:
      image: temporalio/server:latest
      container_name: temporal-frontend-2
      environment:
        - SERVICES=frontend
        - BIND_ON_IP=0.0.0.0
        - TEMPORAL_BROADCAST_ADDRESS=frontend-2
        - NUM_HISTORY_SHARDS=128
        - DB=postgres12
        - POSTGRES_SEEDS=postgres
        - POSTGRES_USER=temporal
        - POSTGRES_PWD=temporal
      networks:
        - temporal-network
      depends_on:
        - postgres
    
    # History instances
    history-1:
      image: temporalio/server:latest
      container_name: temporal-history-1
      environment:
        - SERVICES=history
        - BIND_ON_IP=0.0.0.0
        - TEMPORAL_BROADCAST_ADDRESS=history-1
        - NUM_HISTORY_SHARDS=128
        - DB=postgres12
        - POSTGRES_SEEDS=postgres
        - POSTGRES_USER=temporal
        - POSTGRES_PWD=temporal
      networks:
        - temporal-network
      depends_on:
        - postgres
    
    history-2:
      image: temporalio/server:latest
      container_name: temporal-history-2
      environment:
        - SERVICES=history
        - BIND_ON_IP=0.0.0.0
        - TEMPORAL_BROADCAST_ADDRESS=history-2
        - NUM_HISTORY_SHARDS=128
        - DB=postgres12
        - POSTGRES_SEEDS=postgres
        - POSTGRES_USER=temporal
        - POSTGRES_PWD=temporal
      networks:
        - temporal-network
      depends_on:
        - postgres
    
    history-3:
      image: temporalio/server:latest
      container_name: temporal-history-3
      environment:
        - SERVICES=history
        - BIND_ON_IP=0.0.0.0
        - TEMPORAL_BROADCAST_ADDRESS=history-3
        - NUM_HISTORY_SHARDS=128
        - DB=postgres12
        - POSTGRES_SEEDS=postgres
        - POSTGRES_USER=temporal
        - POSTGRES_PWD=temporal
      networks:
        - temporal-network
      depends_on:
        - postgres
    
    # Matching instances
    matching-1:
      image: temporalio/server:latest
      container_name: temporal-matching-1
      environment:
        - SERVICES=matching
        - BIND_ON_IP=0.0.0.0
        - TEMPORAL_BROADCAST_ADDRESS=matching-1
        - NUM_HISTORY_SHARDS=128
        - DB=postgres12
        - POSTGRES_SEEDS=postgres
        - POSTGRES_USER=temporal
        - POSTGRES_PWD=temporal
      networks:
        - temporal-network
      depends_on:
        - postgres
    
    matching-2:
      image: temporalio/server:latest
      container_name: temporal-matching-2
      environment:
        - SERVICES=matching
        - BIND_ON_IP=0.0.0.0
        - TEMPORAL_BROADCAST_ADDRESS=matching-2
        - NUM_HISTORY_SHARDS=128
        - DB=postgres12
        - POSTGRES_SEEDS=postgres
        - POSTGRES_USER=temporal
        - POSTGRES_PWD=temporal
      networks:
        - temporal-network
      depends_on:
        - postgres
    
    # Worker instance
    worker:
      image: temporalio/server:latest
      container_name: temporal-worker
      environment:
        - SERVICES=worker
        - BIND_ON_IP=0.0.0.0
        - TEMPORAL_BROADCAST_ADDRESS=worker
        - NUM_HISTORY_SHARDS=128
        - DB=postgres12
        - POSTGRES_SEEDS=postgres
        - POSTGRES_USER=temporal
        - POSTGRES_PWD=temporal
      networks:
        - temporal-network
      depends_on:
        - postgres
    
    # Load balancer for Frontend
    nginx:
      image: nginx:alpine
      container_name: temporal-nginx
      ports:
        - "7233:7233"
      volumes:
        - ./nginx.conf:/etc/nginx/nginx.conf:ro
      networks:
        - temporal-network
      depends_on:
        - frontend-1
        - frontend-2
    
    postgres:
      image: postgres:13
      container_name: temporal-postgres
      environment:
        POSTGRES_USER: temporal
        POSTGRES_PASSWORD: temporal
      networks:
        - temporal-network
      volumes:
        - postgres-data:/var/lib/postgresql/data

  networks:
    temporal-network:
      driver: bridge

  volumes:
    postgres-data:
  ```

  ```nginx nginx.conf theme={null}
  stream {
      upstream temporal_frontend {
          least_conn;
          server frontend-1:7233;
          server frontend-2:7233;
      }
      
      server {
          listen 7233;
          proxy_pass temporal_frontend;
      }
  }
  ```
</CodeGroup>

### Kubernetes Deployment

For Kubernetes, use the official Helm charts:

```bash theme={null}
# Add Temporal Helm repository
helm repo add temporalio https://go.temporal.io/helm-charts
helm repo update

# Create namespace
kubectl create namespace temporal

# Install with custom values
helm install temporal temporalio/temporal \
  --namespace temporal \
  --values values.yaml
```

<Accordion title="Example Helm Values">
  ```yaml values.yaml theme={null}
  server:
    replicaCount: 1
    
    config:
      persistence:
        default:
          sql:
            driver: "postgres"
            host: postgres.temporal.svc.cluster.local
            port: 5432
            database: temporal
            user: temporal
            password: temporal
            maxConns: 20
            maxIdleConns: 20
        visibility:
          sql:
            driver: "postgres"
            host: postgres.temporal.svc.cluster.local
            port: 5432
            database: temporal_visibility
            user: temporal
            password: temporal
            maxConns: 10
            maxIdleConns: 10
      
      numHistoryShards: 512

  frontend:
    replicaCount: 3
    resources:
      requests:
        cpu: 500m
        memory: 1Gi
      limits:
        cpu: 2000m
        memory: 2Gi
    service:
      type: LoadBalancer
      port: 7233

  history:
    replicaCount: 5
    resources:
      requests:
        cpu: 1000m
        memory: 2Gi
      limits:
        cpu: 4000m
        memory: 4Gi

  matching:
    replicaCount: 3
    resources:
      requests:
        cpu: 500m
        memory: 1Gi
      limits:
        cpu: 2000m
        memory: 2Gi

  worker:
    replicaCount: 1
    resources:
      requests:
        cpu: 500m
        memory: 1Gi
      limits:
        cpu: 2000m
        memory: 2Gi

  prometheus:
    enabled: true

  grafana:
    enabled: true
  ```
</Accordion>

## Multi-Cluster Replication

For geo-distributed deployments, Temporal supports multi-cluster replication.

### Cluster Configuration

<Tabs>
  <Tab title="Cluster A">
    ```yaml config/cluster-a.yaml theme={null}
    persistence:
      numHistoryShards: 512

    clusterMetadata:
      enableGlobalNamespace: true
      failoverVersionIncrement: 100
      masterClusterName: "cluster-a"
      currentClusterName: "cluster-a"
      clusterInformation:
        cluster-a:
          enabled: true
          initialFailoverVersion: 1
          rpcName: "frontend"
          rpcAddress: "cluster-a.example.com:7233"
          httpAddress: "cluster-a.example.com:7243"
        cluster-b:
          enabled: true
          initialFailoverVersion: 2
          rpcName: "frontend"
          rpcAddress: "cluster-b.example.com:7233"
          httpAddress: "cluster-b.example.com:7243"

    dcRedirectionPolicy:
      policy: "all-apis-forwarding"
    ```
  </Tab>

  <Tab title="Cluster B">
    ```yaml config/cluster-b.yaml theme={null}
    persistence:
      numHistoryShards: 512

    clusterMetadata:
      enableGlobalNamespace: true
      failoverVersionIncrement: 100
      masterClusterName: "cluster-a"
      currentClusterName: "cluster-b"
      clusterInformation:
        cluster-a:
          enabled: true
          initialFailoverVersion: 1
          rpcName: "frontend"
          rpcAddress: "cluster-a.example.com:7233"
          httpAddress: "cluster-a.example.com:7243"
        cluster-b:
          enabled: true
          initialFailoverVersion: 2
          rpcName: "frontend"
          rpcAddress: "cluster-b.example.com:7233"
          httpAddress: "cluster-b.example.com:7243"

    dcRedirectionPolicy:
      policy: "all-apis-forwarding"
    ```
  </Tab>
</Tabs>

### Registering Remote Clusters

After both clusters are running:

```bash theme={null}
# From Cluster A, register Cluster B
tctl --address cluster-a.example.com:7233 \
  admin cluster upsert-remote-cluster \
  --frontend_address "cluster-b.example.com:7233"

# From Cluster B, register Cluster A
tctl --address cluster-b.example.com:7233 \
  admin cluster upsert-remote-cluster \
  --frontend_address "cluster-a.example.com:7233"
```

### Global Namespaces

Create a namespace that replicates across clusters:

```bash theme={null}
tctl --address cluster-a.example.com:7233 \
  namespace register \
  --global \
  --namespace my-global-namespace \
  --clusters cluster-a cluster-b \
  --active_cluster cluster-a
```

### Failover

Manual failover to switch active cluster:

```bash theme={null}
tctl --address cluster-a.example.com:7233 \
  namespace update \
  --namespace my-global-namespace \
  --active_cluster cluster-b
```

## Load Balancing

### Frontend Load Balancing

<Tabs>
  <Tab title="Nginx">
    ```nginx theme={null}
    stream {
        upstream temporal_grpc {
            least_conn;
            server frontend-1.temporal.svc.cluster.local:7233 max_fails=3 fail_timeout=30s;
            server frontend-2.temporal.svc.cluster.local:7233 max_fails=3 fail_timeout=30s;
            server frontend-3.temporal.svc.cluster.local:7233 max_fails=3 fail_timeout=30s;
        }
        
        server {
            listen 7233;
            proxy_pass temporal_grpc;
            proxy_connect_timeout 5s;
            proxy_timeout 300s;
        }
    }
    ```
  </Tab>

  <Tab title="HAProxy">
    ```
    defaults
        mode tcp
        timeout connect 5s
        timeout client 300s
        timeout server 300s

    frontend temporal_grpc
        bind *:7233
        default_backend temporal_backend

    backend temporal_backend
        balance leastconn
        option tcp-check
        server frontend-1 frontend-1:7233 check
        server frontend-2 frontend-2:7233 check
        server frontend-3 frontend-3:7233 check
    ```
  </Tab>

  <Tab title="Kubernetes Service">
    ```yaml theme={null}
    apiVersion: v1
    kind: Service
    metadata:
      name: temporal-frontend
      namespace: temporal
    spec:
      type: LoadBalancer
      sessionAffinity: None
      selector:
        app: temporal-frontend
      ports:
        - name: grpc
          protocol: TCP
          port: 7233
          targetPort: 7233
        - name: http
          protocol: TCP
          port: 7243
          targetPort: 7243
    ```
  </Tab>
</Tabs>

## High Availability Checklist

<Steps>
  <Step title="Multiple Instances">
    * [ ] At least 2 Frontend instances
    * [ ] At least 3 History instances
    * [ ] At least 2 Matching instances
    * [ ] At least 1 Worker instance
  </Step>

  <Step title="Database HA">
    * [ ] Database replication configured
    * [ ] Automatic failover enabled
    * [ ] Regular backups scheduled
    * [ ] Connection pooling tuned
  </Step>

  <Step title="Network Configuration">
    * [ ] Load balancer for Frontend service
    * [ ] Health checks configured
    * [ ] Network policies in place
    * [ ] DNS properly configured
  </Step>

  <Step title="Monitoring">
    * [ ] Prometheus metrics collection
    * [ ] Grafana dashboards
    * [ ] Alerting rules configured
    * [ ] Log aggregation setup
  </Step>

  <Step title="Resource Management">
    * [ ] CPU/Memory limits set
    * [ ] Pod/container auto-scaling
    * [ ] Disk space monitoring
    * [ ] Rate limiting configured
  </Step>
</Steps>

## Troubleshooting Cluster Issues

<AccordionGroup>
  <Accordion title="Shard ownership conflicts" icon="triangle-exclamation">
    **Symptoms**: History service logs show shard ownership conflicts

    **Causes**:

    * Split-brain scenarios
    * Network partitions
    * Clock skew between nodes

    **Solutions**:

    * Ensure NTP is synchronized across all nodes
    * Check network connectivity between services
    * Verify database connectivity
    * Review `maxJoinDuration` setting
  </Accordion>

  <Accordion title="Uneven shard distribution" icon="triangle-exclamation">
    **Symptoms**: Some History instances have many more shards than others

    **Solutions**:

    * Restart History instances one at a time
    * Wait for membership stabilization (30-60s)
    * Check broadcast addresses are correct
    * Verify all instances can reach the database
  </Accordion>

  <Accordion title="Service discovery failures" icon="triangle-exclamation">
    **Symptoms**: Services cannot find each other

    **Solutions**:

    * Check membership ports are accessible
    * Verify broadcast addresses resolve correctly
    * Check firewall rules
    * Ensure database connectivity for membership table
  </Accordion>

  <Accordion title="Load balancer issues" icon="triangle-exclamation">
    **Symptoms**: Connection failures, timeouts, uneven load

    **Solutions**:

    * Configure health checks properly
    * Use least-connections algorithm
    * Set appropriate timeouts (5s connect, 300s idle)
    * Monitor backend health status
  </Accordion>
</AccordionGroup>

## Performance Optimization

### Shard Count Guidelines

<Tabs>
  <Tab title="Small Scale">
    ```yaml theme={null}
    # < 100 workflows/second
    numHistoryShards: 128
    history:
      replicaCount: 3
    # Result: ~43 shards per instance
    ```
  </Tab>

  <Tab title="Medium Scale">
    ```yaml theme={null}
    # 100-1000 workflows/second
    numHistoryShards: 512
    history:
      replicaCount: 5
    # Result: ~102 shards per instance
    ```
  </Tab>

  <Tab title="Large Scale">
    ```yaml theme={null}
    # > 1000 workflows/second
    numHistoryShards: 4096
    history:
      replicaCount: 10
    # Result: ~410 shards per instance
    ```
  </Tab>
</Tabs>

<Warning>
  **Cannot change shard count after deployment!** Choose carefully based on expected scale.
</Warning>

### Resource Recommendations

| Component | Small               | Medium               | Large                 |
| --------- | ------------------- | -------------------- | --------------------- |
| Frontend  | 2 CPU<br />4 GB RAM | 4 CPU<br />8 GB RAM  | 8 CPU<br />16 GB RAM  |
| History   | 4 CPU<br />8 GB RAM | 8 CPU<br />16 GB RAM | 16 CPU<br />32 GB RAM |
| Matching  | 2 CPU<br />4 GB RAM | 4 CPU<br />8 GB RAM  | 8 CPU<br />16 GB RAM  |
| Worker    | 2 CPU<br />4 GB RAM | 2 CPU<br />4 GB RAM  | 4 CPU<br />8 GB RAM   |

<Note>
  These are starting points. Monitor actual usage and adjust based on workload patterns.
</Note>