> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/temporalio/temporal/llms.txt
> Use this file to discover all available pages before exploring further.

# Workers

> Understanding worker processes, polling, and task execution in Temporal

# Workers

Workers are the compute processes that execute your workflow and activity code. They run in your infrastructure, continuously polling Temporal Server for tasks and executing them using the Temporal SDK.

## Worker Architecture

A worker process is a long-running application that:

1. **Registers** workflow and activity implementations with the SDK
2. **Polls** the Temporal Server for workflow and activity tasks
3. **Executes** tasks by running your code
4. **Reports** results back to the server
5. **Repeats** continuously until shut down

```mermaid theme={null}
graph LR
    A[Worker Process] --> B[Workflow Worker]
    A --> C[Activity Worker]
    A --> D[Local Activity Worker]
    
    B --> E[Poll Workflow Tasks]
    C --> F[Poll Activity Tasks]
    
    E --> G[Execute Workflow Code]
    F --> H[Execute Activity Code]
    
    G --> I[Send Commands]
    H --> J[Send Results]
```

## Polling Mechanism

Workers use long-polling to receive tasks from the server with minimal latency:

### Long-Poll Request

<Steps>
  <Step title="Worker sends poll request">
    Worker makes PollWorkflowTaskQueue or PollActivityTaskQueue RPC to Frontend Service
  </Step>

  <Step title="Frontend routes to Matching Service">
    Request is routed to the Matching Service instance responsible for that task queue
  </Step>

  <Step title="Matching waits for task">
    If no task immediately available, Matching Service holds the connection open (long-poll) waiting for a task
  </Step>

  <Step title="Task arrives">
    When History Service dispatches a task, it's matched with the waiting poller
  </Step>

  <Step title="Task returned to worker">
    Matching Service returns the task details and closes the poll request
  </Step>
</Steps>

```go theme={null}
// From service/matching/matcher.go
type TaskMatcher struct {
    // Channels for matching tasks with pollers
    taskC      chan *internalTask  // Regular tasks
    queryTaskC chan *internalTask  // Query tasks
    
    // Track when pollers were last seen
    lastPoller atomic.Int64  // unix nanos of most recent poll
    
    // Rate limiting for task dispatch
    rateLimiter quotas.RateLimiter
}

// Offer matches a task with a waiting poller
func (tm *TaskMatcher) Offer(ctx context.Context, task *internalTask) (bool, error) {
    select {
    case tm.taskC <- task:  // Poller waiting, immediate match
        return true, nil
    default:
        // No poller available, try forwarding to parent partition
        return tm.fwdr.ForwardTask(ctx, task)
    }
}
```

### Sync Match vs Async Match

<CardGroup cols={2}>
  <Card title="Sync Match" icon="bolt">
    **Instant delivery**: Task dispatched directly to waiting poller without touching database. Lowest latency path (\~1ms).
  </Card>

  <Card title="Async Match" icon="database">
    **Queued delivery**: No poller available, task written to database and retrieved later when poller arrives. Higher latency but ensures delivery.
  </Card>
</CardGroup>

## Workflow Task Execution

When a worker receives a workflow task:

### Execution Flow

<Steps>
  <Step title="Worker receives task">
    Task contains workflow execution history (all events since start)
  </Step>

  <Step title="SDK replays history">
    SDK feeds history events through workflow code to reconstruct current state. Must be deterministic.
  </Step>

  <Step title="Workflow code executes">
    Code runs until blocked (waiting for activity, timer, signal) or completes
  </Step>

  <Step title="Generate commands">
    SDK collects commands like ScheduleActivityTask, StartTimer, CompleteWorkflowExecution
  </Step>

  <Step title="Send response">
    Worker sends RespondWorkflowTaskCompleted with command list to History Service
  </Step>
</Steps>

<Accordion title="Workflow Task Response Structure">
  ```go theme={null}
  type RespondWorkflowTaskCompletedRequest struct {
      TaskToken []byte      // Identifies which task this is completing
      Commands  []*Command  // List of commands to execute
      
      // Possible commands:
      // - ScheduleActivityTask
      // - StartTimer 
      // - CompleteWorkflowExecution
      // - FailWorkflowExecution
      // - CancelTimer
      // - RequestCancelActivity
      // - StartChildWorkflowExecution
      // - SignalExternalWorkflowExecution
      // - ContinueAsNewWorkflowExecution
      // - and more...
  }
  ```
</Accordion>

### Workflow Task Failures

Workflow tasks can fail for several reasons:

<CardGroup cols={2}>
  <Card title="Non-Determinism" icon="shuffle">
    Workflow code produced different commands during replay. SDK detects this and fails the task.
  </Card>

  <Card title="Worker Crash" icon="circle-xmark">
    Worker dies during execution. Task timeout fires and task is retried on another worker.
  </Card>

  <Card title="Transient Error" icon="triangle-exclamation">
    Temporary failure like network issue. Task automatically retried with backoff.
  </Card>

  <Card title="Bad Code" icon="bug">
    Unhandled exception in workflow code. Can be configured to fail workflow or retry task.
  </Card>
</CardGroup>

## Activity Task Execution

Activity execution is simpler than workflow execution:

<Steps>
  <Step title="Worker receives activity task">
    Task contains activity input, attempt number, and heartbeat details (if resuming)
  </Step>

  <Step title="Execute activity function">
    Activity code runs with full access to I/O, databases, APIs, etc. No determinism required.
  </Step>

  <Step title="Send heartbeats (optional)">
    Long-running activities periodically heartbeat to prove they're alive
  </Step>

  <Step title="Return result or error">
    Worker sends RespondActivityTaskCompleted or RespondActivityTaskFailed
  </Step>
</Steps>

```go theme={null}
// Activity execution is straightforward
func (w *Worker) executeActivity(task *ActivityTask) {
    // Get registered activity function
    activityFn := w.registry.GetActivity(task.ActivityType)
    
    // Execute with timeout and context
    ctx, cancel := context.WithTimeout(context.Background(), task.StartToCloseTimeout)
    defer cancel()
    
    result, err := activityFn(ctx, task.Input)
    
    // Report result back to server
    if err != nil {
        w.client.RespondActivityTaskFailed(ctx, task.TaskToken, err)
    } else {
        w.client.RespondActivityTaskCompleted(ctx, task.TaskToken, result)
    }
}
```

## Worker Configuration

Workers have several important configuration parameters:

### Concurrency Settings

<Accordion title="Max Concurrent Workflow Tasks">
  Controls how many workflow tasks can execute simultaneously. Each task runs in its own goroutine/thread.

  **Default**: 100
  **Considerations**: Workflow tasks are typically CPU-bound (replay logic). Scale based on CPU cores.
</Accordion>

<Accordion title="Max Concurrent Activity Tasks">
  Controls how many activity tasks can execute simultaneously.

  **Default**: 100
  **Considerations**: Activities often wait on I/O. Can typically be higher than workflow task concurrency.
</Accordion>

<Accordion title="Max Concurrent Local Activity Tasks">
  Controls local activity execution parallelism.

  **Default**: 100
  **Considerations**: Local activities execute inline with workflow task, sharing its resources.
</Accordion>

### Polling Configuration

```go theme={null}
type WorkerOptions struct {
    MaxConcurrentWorkflowTaskPollers int  // Number of parallel pollers
    MaxConcurrentActivityTaskPollers int  // Number of parallel pollers
    
    WorkflowPollersCount int  // Deprecated, use above
    ActivityPollersCount int  // Deprecated, use above
    
    // How long to wait for task before considering poll timed out
    WorkflowTaskTimeout time.Duration
    
    // Other options...
}
```

<Note>
  More pollers increases throughput but also increases load on Matching Service. Typical values: 2-10 pollers per worker process.
</Note>

## Worker Identity and Tracking

Each worker has an identity used for observability:

* **Worker Identity**: Unique identifier (hostname + process ID + UUID)
* **Binary Checksum**: Hash of worker binary for detecting bad deployments
* **Build ID**: Worker version for routing workflows to compatible workers

```go theme={null}
// Server tracks which worker executed each task
type ActivityInfo struct {
    StartedIdentity         string  // Which worker started this attempt
    RetryLastWorkerIdentity string  // Which worker tried last attempt
    LastWorkerDeploymentVersion *deploymentpb.WorkerDeploymentVersion
    // ...
}
```

## Sticky Execution

For performance, Temporal uses "sticky execution" to route workflow tasks back to the same worker:

### How Sticky Execution Works

<Steps>
  <Step title="Worker caches workflow state">
    After completing a workflow task, worker keeps workflow state in memory
  </Step>

  <Step title="Worker specifies sticky task queue">
    Response includes a sticky task queue name unique to this worker
  </Step>

  <Step title="Next task routed to sticky queue">
    History Service dispatches next workflow task to sticky queue first
  </Step>

  <Step title="Fast path if worker available">
    Same worker picks up task, skips replay, resumes from cached state
  </Step>

  <Step title="Fallback to normal queue">
    If worker unavailable after timeout, task dispatched to normal queue
  </Step>
</Steps>

<Accordion title="Sticky Execution Benefits">
  **Performance**: Skip replaying history (can save 100ms+ for large histories)

  **Reduced Server Load**: No need to fetch and transmit full history

  **Lower Latency**: Workflow tasks complete faster, workflows make progress quicker

  **Tradeoff**: Workflow execution pinned to specific worker. If worker dies, new worker must do full replay.
</Accordion>

## Worker Failure Modes

<CardGroup cols={2}>
  <Card title="Worker Process Crash" icon="server">
    **Impact**: In-flight tasks timeout and retry on other workers. Sticky cache lost, requiring history replay.

    **Mitigation**: Workflow tasks automatically redistributed. Minimal impact if worker pool is healthy.
  </Card>

  <Card title="Worker Hangs" icon="stopwatch">
    **Impact**: Tasks never complete, timeouts fire, tasks retried elsewhere.

    **Mitigation**: Set appropriate workflow task timeout. Monitor task latency.
  </Card>

  <Card title="Network Partition" icon="network-wired">
    **Impact**: Worker can't reach server, can't poll for tasks or report results.

    **Mitigation**: Workers retry with backoff. Tasks timeout and retry on healthy workers.
  </Card>

  <Card title="Bad Deployment" icon="triangle-exclamation">
    **Impact**: All workers deploy broken code, all tasks fail.

    **Mitigation**: Binary checksum tracking, gradual rollouts, automated rollback.
  </Card>
</CardGroup>

## Worker Versioning

Temporal supports routing tasks based on worker version:

### Build ID-Based Versioning

* Workers register with a Build ID (e.g., git commit, version number)
* Workflows can be pinned to specific Build ID sets
* New workflows use latest Build ID
* Running workflows stay on their Build ID or migrate per rules

<Tip>
  Use worker versioning for safe, gradual deployments of workflow code changes without disrupting running workflows.
</Tip>

## Monitoring Workers

Key metrics to monitor:

* **Poll Success Rate**: Are workers successfully getting tasks?
* **Task Execution Latency**: How long do tasks take?
* **Task Failure Rate**: How many tasks are failing?
* **Worker Count**: How many workers are active?
* **Sticky Cache Hit Rate**: How often is sticky execution working?
* **Queue Backlog**: Are tasks piling up?

## Related Concepts

* [Task Queues](/concepts/task-queues) - How workers discover tasks
* [Workflows](/concepts/workflows) - What workers execute
* [Activities](/concepts/activities) - Activity execution in workers