> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/temporalio/temporal/llms.txt
> Use this file to discover all available pages before exploring further.

# Metrics Reference

> Complete reference of Temporal Server metrics with descriptions and usage

This page provides a comprehensive reference of all metrics emitted by Temporal Server.

## Metric Types

Temporal uses four metric types:

* **Counter** - Monotonically increasing value (e.g., request count)
* **Gauge** - Point-in-time value that can increase or decrease (e.g., queue size)
* **Timer** - Duration measurements with histogram distribution
* **Histogram** - Value distribution (bytes, dimensionless counts)

## Common Metric Tags

All metrics may include these tags:

| Tag             | Description                  | Values                                               |
| --------------- | ---------------------------- | ---------------------------------------------------- |
| `operation`     | API method or operation name | Method-specific                                      |
| `service_role`  | Service type                 | `frontend`, `history`, `matching`, `worker`, `admin` |
| `namespace`     | Namespace name               | User-defined                                         |
| `task_category` | Task category type           | `transfer`, `timer`, `visibility`, `archival`        |
| `task_type`     | Specific task type           | Task-specific                                        |
| `failure`       | Failure reason               | Error-specific                                       |
| `error_type`    | Error classification         | `invalid_argument`, `not_found`, etc.                |
| `cache_type`    | Cache identifier             | `mutablestate`, `events`, etc.                       |
| `db_kind`       | Database type                | `cassandra`, `mysql`, `postgres`, `sqlite`           |

## Service-Level Metrics

### Request Metrics

#### `service_requests`

**Type:** Counter\
**Description:** Total number of RPC requests received by the service\
**Tags:** `operation`, `service_role`

#### `service_pending_requests`

**Type:** Gauge\
**Description:** Current number of pending/in-flight requests\
**Tags:** `operation`, `service_role`

#### `service_latency`

**Type:** Timer\
**Description:** End-to-end request latency including user code execution\
**Tags:** `operation`, `service_role`\
**Unit:** milliseconds

#### `service_latency_nouserlatency`

**Type:** Timer\
**Description:** Server-side processing latency excluding user code execution\
**Tags:** `operation`, `service_role`\
**Unit:** milliseconds

#### `service_latency_userlatency`

**Type:** Timer\
**Description:** Time spent in user code (workflow/activity execution)\
**Tags:** `operation`, `service_role`\
**Unit:** milliseconds

### Error Metrics

#### `service_errors`

**Type:** Counter\
**Description:** Total unexpected service errors\
**Tags:** `operation`, `service_role`

#### `service_error_with_type`

**Type:** Counter\
**Description:** Service errors categorized by error type\
**Tags:** `operation`, `service_role`, `error_type`

#### `service_errors_invalid_argument`

**Type:** Counter\
**Description:** Requests rejected due to invalid arguments\
**Tags:** `operation`, `service_role`

#### `service_errors_namespace_not_active`

**Type:** Counter\
**Description:** Requests to inactive namespaces\
**Tags:** `operation`, `service_role`, `namespace`

#### `service_errors_resource_exhausted`

**Type:** Counter\
**Description:** Requests rejected due to resource limits\
**Tags:** `operation`, `service_role`, `resource_exhausted_cause`, `resource_exhausted_scope`

#### `service_errors_entity_not_found`

**Type:** Counter\
**Description:** Requests for non-existent entities (workflow, namespace, etc.)\
**Tags:** `operation`, `service_role`

#### `service_errors_execution_already_started`

**Type:** Counter\
**Description:** Attempts to start already-running workflows\
**Tags:** `operation`, `service_role`

#### `service_errors_context_timeout`

**Type:** Counter\
**Description:** Requests that timed out\
**Tags:** `operation`, `service_role`

#### `service_errors_unauthorized`

**Type:** Counter\
**Description:** Authorization denied requests\
**Tags:** `operation`, `service_role`, `namespace`

#### `service_errors_authorize_failed`

**Type:** Counter\
**Description:** Authorization system failures\
**Tags:** `operation`, `service_role`, `namespace`

### Connection Metrics

#### `service_grpc_conn_accepted`

**Type:** Counter\
**Description:** Number of gRPC TCP connections accepted\
**Tags:** `service_role`

#### `service_grpc_conn_closed`

**Type:** Counter\
**Description:** Number of gRPC TCP connections closed\
**Tags:** `service_role`

#### `service_grpc_conn_active`

**Type:** Gauge\
**Description:** Current number of active gRPC connections\
**Tags:** `service_role`

#### `service_dial_latency`

**Type:** Timer\
**Description:** Latency of establishing new TCP connections\
**Tags:** `service_role`\
**Unit:** milliseconds

#### `service_dial_success`

**Type:** Counter\
**Description:** Successful TCP connection attempts\
**Tags:** `service_role`

#### `service_dial_error`

**Type:** Counter\
**Description:** Failed TCP connection attempts\
**Tags:** `service_role`

## Persistence Metrics

All persistence metrics follow the pattern: `Persistence{Operation}Scope`

### Shard Operations

#### `GetOrCreateShard`

**Description:** Shard metadata retrieval or initialization\
**Tags:** `db_kind`

#### `UpdateShard`

**Description:** Shard metadata updates\
**Tags:** `db_kind`

#### `AssertShardOwnership`

**Description:** Shard ownership validation\
**Tags:** `db_kind`

### Workflow Execution Operations

#### `CreateWorkflowExecution`

**Description:** New workflow execution creation\
**Tags:** `db_kind`

#### `GetWorkflowExecution`

**Description:** Workflow execution state retrieval\
**Tags:** `db_kind`

#### `UpdateWorkflowExecution`

**Description:** Workflow execution state updates\
**Tags:** `db_kind`

#### `DeleteWorkflowExecution`

**Description:** Workflow execution deletion\
**Tags:** `db_kind`

#### `GetCurrentExecution`

**Description:** Current workflow run retrieval\
**Tags:** `db_kind`

### History Operations

#### `AppendHistoryNodes`

**Description:** History event append operations\
**Tags:** `db_kind`

#### `ReadHistoryBranch`

**Description:** History event reads\
**Tags:** `db_kind`

#### `ReadHistoryBranchReverse`

**Description:** Reverse chronological history reads\
**Tags:** `db_kind`

#### `DeleteHistoryBranch`

**Description:** History branch deletion\
**Tags:** `db_kind`

#### `ForkHistoryBranch`

**Description:** History branch creation for reset\
**Tags:** `db_kind`

### Task Queue Operations

#### `GetTransferTasks`

**Description:** Transfer task retrieval\
**Tags:** `db_kind`

#### `CompleteTransferTask`

**Description:** Transfer task completion\
**Tags:** `db_kind`

#### `GetTimerTasks`

**Description:** Timer task retrieval\
**Tags:** `db_kind`

#### `CompleteTimerTask`

**Description:** Timer task completion\
**Tags:** `db_kind`

#### `GetVisibilityTasks`

**Description:** Visibility task retrieval\
**Tags:** `db_kind`

#### `GetReplicationTasks`

**Description:** Replication task retrieval\
**Tags:** `db_kind`

### DLQ Operations

#### `EnqueueMessageToDLQ`

**Description:** Messages moved to dead letter queue\
**Tags:** `db_kind`, `queue_type`

#### `ReadMessagesFromDLQ`

**Description:** DLQ message reads\
**Tags:** `db_kind`, `queue_type`

#### `DeleteMessageFromDLQ`

**Description:** Individual DLQ message deletion\
**Tags:** `db_kind`, `queue_type`

#### `RangeDeleteMessagesFromDLQ`

**Description:** Bulk DLQ message deletion\
**Tags:** `db_kind`, `queue_type`

#### `dlq_message_count`

**Type:** Gauge\
**Description:** Number of messages in DLQ by task category\
**Tags:** `task_category`\
**Note:** Emitted every 3 hours from shard 1 owner

### Namespace Operations

#### `CreateNamespace`

**Description:** Namespace creation\
**Tags:** `db_kind`

#### `GetNamespace`

**Description:** Namespace metadata retrieval\
**Tags:** `db_kind`

#### `UpdateNamespace`

**Description:** Namespace metadata updates\
**Tags:** `db_kind`

#### `ListNamespaces`

**Description:** Namespace listing\
**Tags:** `db_kind`

## History Service Metrics

### Workflow Operations

#### `StartWorkflowExecution`

**Description:** Workflow start requests\
**Tags:** `namespace`

#### `RespondWorkflowTaskCompleted`

**Description:** Workflow task completion processing\
**Tags:** `namespace`

#### `RespondActivityTaskCompleted`

**Description:** Activity task completion processing\
**Tags:** `namespace`

#### `RecordActivityTaskHeartbeat`

**Description:** Activity heartbeat processing\
**Tags:** `namespace`

### Shard Management

#### `ShardController`

**Description:** Shard controller operations\
**Tags:** `shard-id`

#### `ShardInfo`

**Description:** Shard metadata operations\
**Tags:** `shard-id`

### Task Processing

#### `TransferQueueProcessor`

**Description:** Transfer queue processing\
**Tags:** `task_type`

#### `TimerQueueProcessor`

**Description:** Timer queue processing\
**Tags:** `task_type`

#### `VisibilityQueueProcessor`

**Description:** Visibility queue processing\
**Tags:** `task_type`

#### `ArchivalQueueProcessor`

**Description:** Archival queue processing\
**Tags:** `task_type`

### Cache Metrics

#### `HistoryCacheGetOrCreate`

**Description:** History cache operations\
**Tags:** `namespace`

#### `EventsCacheGetEvent`

**Description:** Event cache retrievals\
**Tags:** No additional tags

#### `EventsCachePutEvent`

**Description:** Event cache insertions\
**Tags:** No additional tags

#### `EventsCacheGetFromStore`

**Description:** Event cache misses requiring database fetch\
**Tags:** No additional tags

## Matching Service Metrics

#### `PollWorkflowTaskQueue`

**Description:** Workflow task poll operations\
**Tags:** `namespace`

#### `PollActivityTaskQueue`

**Description:** Activity task poll operations\
**Tags:** `namespace`

#### `AddActivityTask`

**Description:** Activity task additions to queue\
**Tags:** `namespace`

#### `AddWorkflowTask`

**Description:** Workflow task additions to queue\
**Tags:** `namespace`

#### `TaskQueueMgr`

**Description:** Task queue manager operations\
**Tags:** `namespace`

## Worker Service Metrics

#### `HistoryArchiver`

**Description:** History archival operations\
**Tags:** `namespace`

#### `VisibilityArchiver`

**Description:** Visibility archival operations\
**Tags:** `namespace`

#### `Replicator`

**Description:** Cross-cluster replication operations\
**Tags:** `namespace`, `cluster`

## Authorization Metrics

#### `service_authorization_latency`

**Type:** Timer\
**Description:** Authorization check duration\
**Tags:** `operation`, `namespace`\
**Unit:** milliseconds

## Resource Metrics

### Lock Metrics

#### `lock_requests`

**Type:** Counter\
**Description:** Lock acquisition attempts\
**Tags:** No additional tags

#### `lock_latency`

**Type:** Timer\
**Description:** Time waiting for lock acquisition\
**Tags:** No additional tags\
**Unit:** milliseconds

### Semaphore Metrics

#### `semaphore_requests`

**Type:** Counter\
**Description:** Semaphore acquisition attempts\
**Tags:** No additional tags

#### `semaphore_failures`

**Type:** Counter\
**Description:** Failed semaphore acquisitions\
**Tags:** No additional tags

#### `semaphore_latency`

**Type:** Timer\
**Description:** Time waiting for semaphore acquisition\
**Tags:** No additional tags\
**Unit:** milliseconds

## TLS Metrics

#### `certificates_expired`

**Type:** Gauge\
**Description:** Number of expired TLS certificates\
**Tags:** No additional tags

#### `certificates_expiring`

**Type:** Gauge\
**Description:** Number of certificates expiring within warning window\
**Tags:** No additional tags

## Workflow Metrics

#### `state_transition_count`

**Type:** Histogram\
**Description:** Number of state transitions per workflow task\
**Tags:** `namespace`

#### `history_size`

**Type:** Histogram\
**Description:** Workflow history size in bytes\
**Tags:** `namespace`\
**Unit:** bytes

#### `history_count`

**Type:** Histogram\
**Description:** Number of events in workflow history\
**Tags:** `namespace`

#### `event_blob_size`

**Type:** Histogram\
**Description:** Size of individual event blobs\
**Tags:** `namespace`\
**Unit:** bytes

#### `search_attributes_size`

**Type:** Histogram\
**Description:** Size of search attributes payload\
**Tags:** `namespace`\
**Unit:** bytes

#### `memo_size`

**Type:** Histogram\
**Description:** Size of workflow memo\
**Tags:** `namespace`\
**Unit:** bytes

## Limit Exceeded Metrics

#### `wf_too_many_pending_activities`

**Type:** Counter\
**Description:** Workflow tasks failed due to pending activity limit\
**Tags:** `namespace`

#### `wf_too_many_pending_child_workflows`

**Type:** Counter\
**Description:** Workflow tasks failed due to pending child workflow limit\
**Tags:** `namespace`

#### `wf_too_many_pending_cancel_requests`

**Type:** Counter\
**Description:** Workflow tasks failed due to pending cancel request limit\
**Tags:** `namespace`

#### `blob_size_error`

**Type:** Counter\
**Description:** Requests failed due to blob size limits\
**Tags:** `namespace`

## Client Metrics

#### `client_requests`

**Type:** Counter\
**Description:** Outbound client requests to other services\
**Tags:** `service_role`, `operation`

#### `client_errors`

**Type:** Counter\
**Description:** Client request errors\
**Tags:** `service_role`, `operation`

#### `client_latency`

**Type:** Timer\
**Description:** Client request latency\
**Tags:** `service_role`, `operation`\
**Unit:** milliseconds

## Metric Collection Best Practices

### Histogram Bucket Configuration

Customize histogram buckets based on your workload:

```yaml theme={null}
global:
  metrics:
    clientConfig:
      perUnitHistogramBoundaries:
        # For high-throughput, low-latency workflows
        milliseconds: [1, 2, 5, 10, 25, 50, 100, 250, 500, 1000]
        # For large payload workflows
        bytes: [1024, 10240, 102400, 1048576, 10485760]
```

### Tag Cardinality

Be cautious with high-cardinality tags:

```yaml theme={null}
global:
  metrics:
    clientConfig:
      excludeTags:
        namespace:
          - "whitelisted-namespace-1"
          - "whitelisted-namespace-2"
        workflowType:
          - "important-workflow"
```

### Sampling for High Volume

For extremely high request rates, consider sampling at the metrics backend level rather than in Temporal.

## See Also

* [Monitoring Guide](/operations/monitoring) - Setup and alerting
* [Scaling Guide](/operations/scaling) - Using metrics for scaling decisions
* [Persistence](/operations/persistence) - Database-level metrics