> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/temporalio/temporal/llms.txt
> Use this file to discover all available pages before exploring further.

# Task Queues

> Task queue architecture, partitioning, and routing in Temporal Server

# Task Queues

Task queues are the routing mechanism that connects the Temporal Server to your workers. They act as a bridge between workflow executions in the History Service and worker processes that execute workflow and activity code.

## Task Queue Basics

A task queue is a logical queue that workers poll for tasks. When the History Service needs workflow or activity code to execute, it dispatches tasks to the appropriate task queue.

### Two Types of Tasks

<CardGroup cols={2}>
  <Card title="Workflow Tasks" icon="diagram-project">
    Prompts worker to advance workflow execution by replaying history and generating new commands.
  </Card>

  <Card title="Activity Tasks" icon="list-check">
    Requests worker to execute specific activity function with given input.
  </Card>
</CardGroup>

<Note>
  These "task queue tasks" are user-visible concepts, distinct from internal History Service tasks (Transfer Tasks, Timer Tasks) which are implementation details.
</Note>

## Task Queue Architecture

The Matching Service manages all task queues:

```mermaid theme={null}
graph TB
    subgraph History Service
        H1[History Shard 1]
        H2[History Shard 2]
        H3[History Shard 3]
    end
    
    subgraph Matching Service
        M1[Matching Instance 1]
        M2[Matching Instance 2]
        
        subgraph "Task Queue: 'payments'"
            P0[Partition 0]
            P1[Partition 1]
            P2[Partition 2]
            P3[Partition 3]
        end
    end
    
    subgraph Workers
        W1[Worker 1]
        W2[Worker 2]
        W3[Worker 3]
    end
    
    H1 --> M1
    H2 --> M1
    H3 --> M2
    
    M1 --> P0
    M1 --> P1
    M2 --> P2
    M2 --> P3
    
    P0 --> W1
    P1 --> W2
    P2 --> W3
    P3 --> W1
```

### Task Queue Partitioning

To handle high throughput, each task queue is split into multiple partitions:

```go theme={null}
// From service/matching/task_queue_partition_manager.go
type taskQueuePartitionManagerImpl struct {
    engine    *matchingEngineImpl
    partition tqid.Partition  // Which partition this manages
    ns        *namespace.Namespace
    config    *taskQueueConfig
    
    // Versioned queues for worker versioning
    versionedQueues     map[PhysicalTaskQueueVersion]physicalTaskQueueManager
    versionedQueuesLock sync.RWMutex
    
    userDataManager     userDataManager
    rateLimitManager    *rateLimitManager
    
    // Default queue future for initialization
    defaultQueueFuture *future.FutureImpl[physicalTaskQueueManager]
}
```

<Accordion title="Why Partition Task Queues?">
  **Scalability**: Single queue partition limited by one Matching Service instance. Partitions spread load across instances.

  **Throughput**: Each partition handles subset of tasks. More partitions = higher total throughput.

  **Default**: 4 partitions per task queue, configurable per namespace.

  **Trade-offs**: More partitions increase dispatching overhead. Optimal partition count depends on task rate.
</Accordion>

## Task Dispatching

When History Service schedules a task, it goes through this flow:

<Steps>
  <Step title="History creates Transfer Task">
    Workflow/Activity task decision creates a Transfer Task in History shard's queue
  </Step>

  <Step title="Queue Processor picks up task">
    Background queue processor reads Transfer Task from persistence
  </Step>

  <Step title="Call Matching Service">
    History makes AddWorkflowTask or AddActivityTask RPC to Matching Service
  </Step>

  <Step title="Matching routes to partition">
    Matching Service hashes task queue name to select partition
  </Step>

  <Step title="Attempt sync match">
    If worker is polling, task delivered immediately (sync match)
  </Step>

  <Step title="Or write to database">
    If no poller available, task written to persistence (async match)
  </Step>
</Steps>

```go theme={null}
// From service/history/transfer_queue_active_task_executor.go
func (t *transferQueueActiveTaskExecutor) processWorkflowTask(
    ctx context.Context,
    task *tasks.WorkflowTask,
) error {
    // Load workflow state
    // Build task info
    // Call Matching Service
    return t.matchingClient.AddWorkflowTask(ctx, &matchingservice.AddWorkflowTaskRequest{
        NamespaceId:       task.NamespaceID,
        Execution:         task.WorkflowExecution,
        TaskQueue:         &taskqueuepb.TaskQueue{Name: taskQueueName},
        ScheduledEventId:  task.ScheduledEventID,
        ScheduleToStartTimeout: timeout,
    })
}
```

## Sync Match vs Async Match

The Matching Service optimizes for low latency through sync matching:

### Sync Match (Fast Path)

<Steps>
  <Step title="Worker is already polling">
    Worker has open long-poll connection waiting for task
  </Step>

  <Step title="Task arrives at Matching">
    History Service dispatches task to Matching Service
  </Step>

  <Step title="Instant match">
    Task delivered directly to waiting poller without database write
  </Step>

  <Step title="Low latency">
    Total time from History to Worker: 1-5ms typically
  </Step>
</Steps>

### Async Match (Slow Path)

<Steps>
  <Step title="No poller available">
    Task arrives but no worker is currently polling
  </Step>

  <Step title="Write to database">
    Task persisted to Matching Service database
  </Step>

  <Step title="Worker polls later">
    When worker polls, task read from database and returned
  </Step>

  <Step title="Higher latency">
    Database round-trip adds latency, but ensures task not lost
  </Step>
</Steps>

```go theme={null}
// From service/matching/matcher.go
func (tm *TaskMatcher) Offer(ctx context.Context, task *internalTask) (bool, error) {
    // Check if we should prefer database due to backlog
    if !tm.isBacklogNegligible() {
        return false, nil  // Force async match for ordering
    }
    
    // Try sync match with waiting poller
    select {
    case tm.taskC <- task:  // Poller waiting!
        if task.responseC != nil {
            err := <-task.responseC  // Wait for poller to accept
            return true, err.startErr
        }
        return false, nil
    default:
        // No poller, try forwarding to parent partition
        return tm.fwdr.ForwardTask(ctx, task)
    }
}
```

## Task Queue Forwarding

To improve sync match rate, Matching uses hierarchical forwarding:

### Forwarding Mechanism

```mermaid theme={null}
graph TD
    Root[Root Partition]
    Child1[Partition 1]
    Child2[Partition 2]
    Child3[Partition 3]
    Child4[Partition 4]
    
    Root --> Child1
    Root --> Child2
    Root --> Child3
    Root --> Child4
    
    Child1 -.Task has no local poller.-> Root
    Child2 -.Poller has no local tasks.-> Root
```

<Accordion title="How Forwarding Works">
  **Forward Task to Parent**: Child partition has task but no poller → forwards task to parent/root

  **Forward Poller to Parent**: Child partition has poller but no task → forwards poller to parent/root

  **Match at Root**: Root partition has better chance of matching tasks with pollers from all children

  **Benefits**: Higher sync match rate, better load distribution, faster task delivery

  **Cost**: Additional RPC hops, slightly higher latency for forwarded tasks
</Accordion>

```go theme={null}
// From service/matching/forwarder.go
type Forwarder struct {
    cfg       *forwarderConfig
    partition tqid.Partition
    client    matchingservice.MatchingServiceClient
    
    // Token bucket for rate limiting forwarding
    addReqTokenBucket atomic.Pointer[dynamicconfig.TaskqueueQuotaBucket]
    pollReqTokenBucket atomic.Pointer[dynamicconfig.TaskqueueQuotaBucket]
}

func (fwdr *Forwarder) ForwardTask(ctx context.Context, task *internalTask) error {
    // Get parent partition
    parent := fwdr.partition.Parent()
    
    // Forward to parent's Matching instance
    return fwdr.client.AddWorkflowTask(ctx, &matchingservice.AddWorkflowTaskRequest{
        // ... task details
        ForwardInfo: &matchingservice.ForwardInfo{
            SourcePartition: fwdr.partition.PartitionId(),
        },
    })
}
```

## Task Queue Lifecycle

### Loading

Task queue partitions are loaded on-demand:

1. **First poll or task** for a task queue triggers loading
2. **Partition manager created** for that partition
3. **State loaded from database** (backlog, ack levels, etc.)
4. **Matcher started** to match tasks with pollers

### Unloading

Partitions unload when idle:

* **No pollers** for configured duration (e.g., 5 minutes)
* **No tasks** dispatched
* **Memory pressure** on Matching Service

```go theme={null}
const (
    unloadCauseIdle = "idle"           // No activity
    unloadCauseConfigChange = "config"  // Dynamic config changed
    unloadCauseError = "error"         // Persistent errors
)
```

<Note>
  Unloading is an optimization. When next task/poll arrives, partition reloads automatically. Users don't see interruption.
</Note>

## Task Queue Backlog

When task arrival rate exceeds worker processing rate, backlog accumulates:

### Backlog Management

<Accordion title="Backlog Storage">
  Tasks that can't sync match are written to persistence:

  **Cassandra**: Separate table per task queue partition
  **SQL**: Rows in tasks table with task queue + partition columns

  Backlog is FIFO ordered by task ID (monotonically increasing).
</Accordion>

<Accordion title="Backlog Reading">
  ```go theme={null}
  // From service/matching/task_reader.go
  type taskReader struct {
      tlMgr           taskQueuePartitionManager
      db              *taskQueueDB
      matcher         *TaskMatcher
      
      // Read batches of tasks from database
      taskBatchSize   int
  }

  // Continuously read backlog tasks and offer to matcher
  func (tr *taskReader) dispatchBufferedTasks() {
      for !tr.stopped() {
          tasks := tr.db.ReadTasks(batchSize)  // Read from persistence
          for _, task := range tasks {
              tr.matcher.Offer(task)  // Try to dispatch
          }
      }
  }
  ```
</Accordion>

<Accordion title="Backlog Metrics">
  Key metrics:

  * **Task Queue Backlog Age**: How old is the oldest task?
  * **Task Queue Backlog Size**: How many tasks waiting?
  * **Task Dispatch Latency**: Time from scheduling to worker pickup
</Accordion>

## Task Queue Versioning

For worker versioning, each task queue has multiple physical queues:

```go theme={null}
// From service/matching/task_queue_partition_manager.go
type taskQueuePartitionManagerImpl struct {
    // One versioned queue per Build ID
    versionedQueues map[PhysicalTaskQueueVersion]physicalTaskQueueManager
    
    // User data manager tracks versioning rules
    userDataManager userDataManager
}
```

<CardGroup cols={2}>
  <Card title="Default Queue" icon="inbox">
    Unversioned tasks and workers without Build ID use default queue
  </Card>

  <Card title="Versioned Queues" icon="code-branch">
    Each Build ID gets dedicated queue. Tasks routed by versioning rules.
  </Card>
</CardGroup>

## Task Queue Configuration

### Dynamic Configuration

```go theme={null}
type taskQueueConfig struct {
    // How many partitions for this task queue
    NumReadPartitions int
    NumWritePartitions int
    
    // Task lifecycle
    MaxTaskQueueIdleTime time.Duration  // When to unload
    MaxTaskBatchSize int                // Backlog read batch size
    
    // Rate limiting
    TaskDispatchRPS float64
    
    // Forwarding
    ForwarderMaxChildrenPerNode int
    ForwarderMaxOutstandingPolls int
    ForwarderMaxOutstandingTasks int
}
```

### Rate Limiting

Task queues support rate limiting task dispatch:

* **Global RPS limit** per task queue
* **Smoothed delivery** prevents spikes
* **Shared across partitions**
* **Configurable per namespace**

<Warning>
  Rate limiting applies to task dispatch from Matching to workers, not task arrival from History Service. Tasks buffer in Matching if rate limited.
</Warning>

## Sticky Task Queues

Workers can request "sticky" execution for performance:

* **Normal task queue**: `my-task-queue`
* **Sticky task queue**: `__sticky__:${workerID}`

Sticky queues:

* Route workflow tasks back to same worker
* Enable workflow cache reuse (skip replay)
* Timeout quickly and fallback to normal queue
* Per-worker, not shared across workers

## Task Queue Observability

Key signals to monitor:

### Metrics

* `task_queue_backlog_age_seconds`: Age of oldest task
* `task_queue_backlog_size`: Number of pending tasks
* `poll_success_rate`: Poller success rate
* `sync_match_rate`: Percentage of sync vs async matches
* `task_dispatch_latency`: Time from schedule to worker

### Troubleshooting

<CardGroup cols={2}>
  <Card title="High Backlog Age" icon="clock">
    **Cause**: Not enough workers or workers too slow

    **Solution**: Scale workers, optimize activity code, increase concurrency
  </Card>

  <Card title="Low Sync Match Rate" icon="shuffle">
    **Cause**: Tasks arriving when no pollers waiting

    **Solution**: Increase poller count, reduce partition count, enable forwarding
  </Card>

  <Card title="High Task Latency" icon="hourglass">
    **Cause**: Network issues, Matching overload, database slowness

    **Solution**: Check Matching Service health, database performance, network latency
  </Card>

  <Card title="No Recent Poller" icon="user-slash">
    **Cause**: All workers down or misconfigured

    **Solution**: Verify workers running, check task queue name matches, review logs
  </Card>
</CardGroup>

## Related Concepts

* [Workers](/concepts/workers) - Poll task queues and execute tasks
* [Workflows](/concepts/workflows) - Generate workflow tasks
* [Activities](/concepts/activities) - Generate activity tasks