> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/temporalio/temporal/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitoring and Observability

> Monitor Temporal Server health, performance, and operations using metrics, logs, and traces

Temporal Server provides comprehensive monitoring capabilities through metrics, structured logging, and distributed tracing to observe cluster health and performance.

## Metrics Collection

Temporal emits metrics using either **Prometheus** or **StatsD** backends. The metrics framework can be configured to use either Tally or OpenTelemetry.

### Prometheus Configuration

Configure Prometheus metrics in your `config.yaml`:

```yaml theme={null}
global:
  metrics:
    prometheus:
      framework: "opentelemetry"  # or "tally"
      listenAddress: "127.0.0.1:8000"
      handlerPath: "/metrics"
      loggerRPS: 0  # 0 means no limit
```

**Framework Options:**

* `tally` - Legacy framework using uber-go/tally
* `opentelemetry` - Modern OpenTelemetry-based metrics (recommended)

**Configuration Fields:**

* `listenAddress` - Address where Prometheus scrapes metrics
* `handlerPath` - HTTP endpoint path (default: `/metrics`)
* `loggerRPS` - Rate limit for metric logger (0 = unlimited)

### StatsD Configuration

For StatsD integration:

```yaml theme={null}
global:
  metrics:
    statsd:
      framework: "opentelemetry"
      hostPort: "127.0.0.1:8125"
      prefix: "temporal"
      flushInterval: "1s"
      flushBytes: 1432
      reporter:
        tagSeparator: ":"
```

**Configuration Fields:**

* `hostPort` - StatsD server address
* `prefix` - Metric name prefix
* `flushInterval` - Batch flush interval (default: 1s)
* `flushBytes` - Maximum UDP packet size (default: 1432)
* `tagSeparator` - Character to separate tags (optional)

### Common Metrics Configuration

```yaml theme={null}
global:
  metrics:
    clientConfig:
      tags:
        environment: "production"
        cluster: "us-east-1"
      excludeTags:
        namespace:
          - "system-namespace"  # whitelist only
      prefix: "temporal_"
      perUnitHistogramBoundaries:
        dimensionless: [1, 2, 5, 10, 20, 50, 100, 200, 500, 1000]
        milliseconds: [1, 2, 5, 10, 20, 50, 100, 200, 500, 1000, 2000, 5000, 10000]
        bytes: [1024, 2048, 4096, 8192, 16384, 32768, 65536, 131072, 262144, 524288]
```

**Options:**

* `tags` - Global tags added to all metrics
* `excludeTags` - Filter sensitive tag values (replaced with `_tag_excluded_`)
* `prefix` - Prefix for all metric names
* `perUnitHistogramBoundaries` - Custom histogram buckets by unit type
* `withoutUnitSuffix` - Remove unit suffixes (OpenTelemetry only)
* `withoutCounterSuffix` - Remove `_total` suffix from counters (OpenTelemetry only)
* `recordTimerInSeconds` - Emit timers in seconds instead of milliseconds

## Key Metrics by Service

### Service Health Metrics

These metrics track overall service health:

<CodeGroup>
  ```plaintext Service Request Metrics theme={null}
  service_requests              # Total RPC requests received
  service_pending_requests      # Current pending requests (gauge)
  service_errors                # Unexpected service errors
  service_error_with_type       # Errors by error type
  service_latency               # Request latency
  service_latency_nouserlatency # Server-side latency only
  service_latency_userlatency   # User workflow latency
  ```

  ```plaintext Connection Metrics theme={null}
  service_grpc_conn_accepted  # TCP connections accepted
  service_grpc_conn_closed    # TCP connections closed
  service_grpc_conn_active    # Active TCP connections (gauge)
  service_dial_latency        # Connection establishment time
  service_dial_success        # Successful connection attempts
  service_dial_error          # Failed connection attempts
  ```
</CodeGroup>

**Common Tags:**

* `operation` - API method name
* `service_role` - Service type (frontend, history, matching, worker)

### Persistence Layer Metrics

Track database operations:

```plaintext theme={null}
# Shard Operations
GetOrCreateShard
UpdateShard
AssertShardOwnership

# Workflow Execution
CreateWorkflowExecution
GetWorkflowExecution
UpdateWorkflowExecution
DeleteWorkflowExecution

# Task Queue Operations
CreateTaskQueue
GetTaskQueue
UpdateTaskQueue
DeleteTaskQueue

# Task Operations
GetTransferTasks
CompleteTransferTask
GetTimerTasks
CompleteTimerTask
GetVisibilityTasks
GetReplicationTasks

# History Operations
AppendHistoryNodes
ReadHistoryBranch
DeleteHistoryBranch
```

Each persistence operation emits:

* Request count
* Error count
* Latency histogram
* `db_kind` tag (cassandra, mysql, postgres, sqlite)

### History Service Metrics

```plaintext theme={null}
# Core Operations
StartWorkflowExecution
RecordActivityTaskHeartbeat
RespondWorkflowTaskCompleted
RespondActivityTaskCompleted

# Shard Management
ShardController
ShardInfo

# Task Processing
TransferQueueProcessor
TimerQueueProcessor
VisibilityQueueProcessor
ArchivalQueueProcessor
OutboundQueueProcessor

# Cache Performance
HistoryCacheGetOrCreate
EventsCacheGetEvent
EventsCachePutEvent
```

### Matching Service Metrics

```plaintext theme={null}
PollWorkflowTaskQueue
PollActivityTaskQueue
AddActivityTask
AddWorkflowTask
TaskQueueMgr
TaskQueuePartitionManager
```

### Authorization Metrics

```plaintext theme={null}
service_authorization_latency      # Authorization check duration
service_errors_unauthorized        # Rejected requests
service_errors_authorize_failed    # Authorization system errors
```

Tagged with:

* `namespace` - Target namespace
* `operation` - API being authorized

## Error Tracking

Error metrics by type:

```plaintext theme={null}
service_errors_invalid_argument
service_errors_namespace_not_active
service_errors_resource_exhausted
service_errors_entity_not_found
service_errors_execution_already_started
service_errors_context_timeout
service_errors_retry_task
service_errors_incomplete_history
service_errors_nondeterministic
```

## Resource Metrics

### Lock and Semaphore Usage

```plaintext theme={null}
lock_requests        # Lock acquisition attempts
lock_latency         # Time waiting for locks
semaphore_requests   # Semaphore acquisition attempts
semaphore_failures   # Failed semaphore acquisitions
semaphore_latency    # Time waiting for semaphore
```

### Cache Metrics

```plaintext theme={null}
NamespaceCache
EventsCacheGetEvent
EventsCachePutEvent
EventsCacheGetFromStore
VersionMembershipCacheGet
VersionMembershipCachePut
```

Tagged with `cache_type`:

* `mutablestate`
* `events`
* `version_membership`
* `routing_info`

## TLS Certificate Monitoring

```plaintext theme={null}
certificates_expired   # Number of expired certificates (gauge)
certificates_expiring  # Number of certificates expiring soon (gauge)
```

Configure certificate monitoring:

```yaml theme={null}
global:
  tls:
    expirationChecks:
      warningWindow: "720h"  # 30 days
      errorWindow: "168h"    # 7 days
      checkInterval: "1h"
```

## Alerting Guidelines

### Critical Alerts

Set up alerts for:

1. **Service Availability**
   ```promql theme={null}
   rate(service_errors[5m]) > 0.05  # 5% error rate
   service_pending_requests > 1000
   ```

2. **Persistence Layer**
   ```promql theme={null}
   rate(UpdateShard_errors[1m]) > 0
   histogram_quantile(0.99, rate(GetWorkflowExecution_latency[5m])) > 1000
   ```

3. **Shard Health**
   ```promql theme={null}
   rate(ShardController_errors[5m]) > 0
   ```

4. **Certificate Expiration**
   ```promql theme={null}
   certificates_expired > 0
   certificates_expiring > 0
   ```

### Warning Alerts

1. **High Latency**
   ```promql theme={null}
   histogram_quantile(0.95, rate(service_latency[5m])) > 500
   ```

2. **Resource Pressure**
   ```promql theme={null}
   semaphore_failures > 100
   lock_latency > 100
   ```

3. **Cache Efficiency**
   ```promql theme={null}
   rate(EventsCacheGetFromStore[5m]) / rate(EventsCacheGetEvent[5m]) > 0.5
   ```

## Logging Configuration

Configure structured logging:

```yaml theme={null}
log:
  stdout: true
  level: "info"  # debug, info, warn, error
  outputFile: "/var/log/temporal/server.log"
  encoding: "json"  # json or console
```

**Log Levels:**

* `debug` - Detailed diagnostic information
* `info` - General operational events
* `warn` - Warning messages, degraded state
* `error` - Error events, requires attention

**Important Log Tags:**

* `namespace` - Namespace name
* `workflowID` - Workflow execution ID
* `runID` - Workflow run ID
* `operation` - Operation being performed
* `error` - Error details
* `shard-id` - History shard ID

## Health Checks

Temporal exposes health check endpoints:

```bash theme={null}
# Frontend health
curl http://localhost:7233/health

# Service-specific health
curl http://localhost:7234/health  # History
curl http://localhost:7235/health  # Matching
```

## Distributed Tracing

Enable OpenTelemetry tracing:

```yaml theme={null}
global:
  otel:
    enabled: true
    exporters:
      - type: "otlp"
        endpoint: "otel-collector:4317"
        insecure: false
        headers:
          api-key: "your-api-key"
```

Tracing captures:

* Request flow across services
* Persistence operation timing
* Cross-namespace operations
* Replication latency

## Dashboard Recommendations

### Service Overview Dashboard

* Request rate by service and operation
* Error rate and types
* Latency percentiles (p50, p95, p99)
* Active connections

### Persistence Dashboard

* Operation latency by type
* Error rates by operation
* Connection pool utilization
* Query duration

### Workflow Execution Dashboard

* Workflow start rate
* Workflow completion rate
* Task queue backlog
* Activity timeouts

### Resource Usage Dashboard

* CPU and memory per service
* Lock contention
* Cache hit rates
* GC pause time

## See Also

* [Metrics Reference](/operations/metrics) - Complete metrics catalog
* [Scaling Guide](/operations/scaling) - Performance tuning
* [Security](/operations/security) - Authentication and authorization