Archival Configuration

Configure long-term archival storage for workflow histories and visibility records

Temporal supports archiving workflow histories and visibility records to external storage for compliance, auditing, and long-term retention.

Overview

Archival moves data from the active persistence store to cheaper, long-term storage:

• History Archival - Complete workflow execution history
• Visibility Archival - Workflow metadata for search and list

Archival happens automatically after workflow completion based on namespace configuration.

Supported Providers

Temporal includes built-in archival providers:

• Filestore - Local or network filesystem
• S3 - Amazon S3 and compatible services
• Google Cloud Storage - GCS buckets

Global Configuration

Enable archival cluster-wide:

archival:
  history:
    state: "enabled"  # enabled, disabled
    enableRead: true
    provider:
      filestore:
        fileMode: "0666"
        dirMode: "0766"
      s3store:
        region: "us-east-1"
        endpoint: ""  # Leave empty for AWS S3
        s3ForcePathStyle: false
      gstorage:
        credentialsPath: "/path/to/gcp-credentials.json"
  
  visibility:
    state: "enabled"
    enableRead: true
    provider:
      filestore:
        fileMode: "0666"
        dirMode: "0766"

States:

• enabled - Archival active, namespaces can enable
• disabled - Archival unavailable
• paused - Temporarily suspended

Options:

• enableRead - Allow reading from archive

Filestore Provider

Configuration

archival:
  history:
    state: "enabled"
    enableRead: true
    provider:
      filestore:
        fileMode: "0666"  # File permissions (octal)
        dirMode: "0766"   # Directory permissions (octal)

URI Format

file:///path/to/archive/directory
file:///mnt/nfs/temporal/archive

Best Practices

1. Network Storage: Use NFS or similar for multi-node access
2. Permissions: Ensure all Temporal workers can read/write
3. Quotas: Monitor disk usage, implement rotation
4. Backup: Regular backups of archive directory

Example Setup

# Create archive directory
mkdir -p /mnt/temporal-archive/history
mkdir -p /mnt/temporal-archive/visibility

# Set permissions
chmod 0766 /mnt/temporal-archive/history
chmod 0766 /mnt/temporal-archive/visibility

# Set ownership
chown -R temporal:temporal /mnt/temporal-archive

S3 Provider

Configuration

archival:
  history:
    state: "enabled"
    enableRead: true
    provider:
      s3store:
        region: "us-east-1"
        endpoint: ""  # Custom endpoint for S3-compatible (e.g., MinIO)
        s3ForcePathStyle: false  # Set true for MinIO

URI Format

s3://bucket-name/path/prefix
s3://temporal-archive-prod/history

Authentication

Use IAM roles (recommended) or credentials:

IAM Role (AWS)

# No explicit credentials needed
# Attach IAM role to EC2 instances/EKS pods

IAM Policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::temporal-archive-prod/*",
        "arn:aws:s3:::temporal-archive-prod"
      ]
    }
  ]
}

Credentials

# Set environment variables
export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"
export AWS_SESSION_TOKEN="your-session-token"  # Optional

MinIO

archival:
  history:
    provider:
      s3store:
        region: "us-east-1"  # Required but ignored by MinIO
        endpoint: "http://minio.example.com:9000"
        s3ForcePathStyle: true

Credentials:

export AWS_ACCESS_KEY_ID="minio-access-key"
export AWS_SECRET_ACCESS_KEY="minio-secret-key"

S3 Bucket Configuration

# Create bucket
aws s3 mb s3://temporal-archive-prod

# Enable versioning (recommended)
aws s3api put-bucket-versioning \
  --bucket temporal-archive-prod \
  --versioning-configuration Status=Enabled

# Set lifecycle policy
aws s3api put-bucket-lifecycle-configuration \
  --bucket temporal-archive-prod \
  --lifecycle-configuration file://lifecycle.json

Lifecycle policy (lifecycle.json):

{
  "Rules": [
    {
      "Id": "TransitionToIA",
      "Status": "Enabled",
      "Transitions": [
        {
          "Days": 90,
          "StorageClass": "STANDARD_IA"
        },
        {
          "Days": 365,
          "StorageClass": "GLACIER"
        }
      ]
    }
  ]
}

Google Cloud Storage Provider

Configuration

archival:
  history:
    state: "enabled"
    enableRead: true
    provider:
      gstorage:
        credentialsPath: "/path/to/service-account-key.json"

URI Format

gs://bucket-name/path/prefix
gs://temporal-archive-prod/history

Authentication

Service Account

archival:
  history:
    provider:
      gstorage:
        credentialsPath: "/etc/temporal/gcp-credentials.json"

Create service account:

# Create service account
gcloud iam service-accounts create temporal-archiver \
  --display-name="Temporal Archiver"

# Grant storage permissions
gcloud projects add-iam-policy-binding PROJECT_ID \
  --member="serviceAccount:temporal-archiver@PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/storage.objectAdmin"

# Create key
gcloud iam service-accounts keys create gcp-credentials.json \
  --iam-account=temporal-archiver@PROJECT_ID.iam.gserviceaccount.com

Workload Identity (GKE)

# Use Workload Identity instead of key file
# Link Kubernetes service account to GCP service account

gcloud iam service-accounts add-iam-policy-binding \
  temporal-archiver@PROJECT_ID.iam.gserviceaccount.com \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:PROJECT_ID.svc.id.goog[temporal/temporal-worker]"

GCS Bucket Configuration

# Create bucket
gsutil mb -l us-east1 gs://temporal-archive-prod

# Set lifecycle policy
gsutil lifecycle set lifecycle.json gs://temporal-archive-prod

Lifecycle policy (lifecycle.json):

{
  "lifecycle": {
    "rule": [
      {
        "action": {"type": "SetStorageClass", "storageClass": "NEARLINE"},
        "condition": {"age": 90}
      },
      {
        "action": {"type": "SetStorageClass", "storageClass": "COLDLINE"},
        "condition": {"age": 365}
      }
    ]
  }
}

Namespace Configuration

Configure archival per namespace:

# Create namespace with archival
tctl namespace register production \
  --retention 7 \
  --history-archival-state enabled \
  --history-uri "s3://temporal-archive-prod/history/production" \
  --visibility-archival-state enabled \
  --visibility-uri "s3://temporal-archive-prod/visibility/production"

Or update existing:

tctl namespace update production \
  --history-archival-state enabled \
  --history-uri "s3://temporal-archive-prod/history/production"

Namespace States:

• enabled - Archive workflows in this namespace
• disabled - Do not archive

Default Namespace Archival

Set defaults for new namespaces:

namespaceDefaults:
  archival:
    history:
      state: "disabled"  # disabled, enabled
      URI: "s3://temporal-archive-prod/history"
    visibility:
      state: "disabled"
      URI: "s3://temporal-archive-prod/visibility"

Reading Archived Data

History

Retrieve archived workflow history:

tctl workflow show \
  --workflow-id my-workflow-id \
  --run-id run-id \
  --namespace production

Temporal automatically checks archive if not in primary store.

Visibility

Query archived workflows:

tctl workflow list \
  --namespace production \
  --query 'WorkflowType="MyWorkflow" AND CloseTime > "2024-01-01"'

Note: Visibility queries against archive may be slower than active store.

Archival Process

Archival occurs through system workflows:

1. Workflow Completes - Workflow reaches final state
2. Retention Period - Waits for namespace retention period
3. Archival Task - System worker picks up archival task
4. Upload History - History events uploaded to archive URI
5. Upload Visibility - Visibility record uploaded
6. Delete from DB - Primary store data deleted

Archival Workers

System workers handle archival:

services:
  worker:
    rpc:
      grpcPort: 7239
      membershipPort: 6939

Scale workers based on archival throughput:

# Monitor archival queue size
histogram_quantile(0.99, rate(ArchivalQueueProcessor_latency_bucket[5m]))

Monitoring Archival

Metrics

HistoryArchiver       # History archival operations
VisibilityArchiver    # Visibility archival operations
ArchiverClient        # Archival client operations
ArchivalQueueProcessor # Archival task processing

Each emits:

• Request count
• Error count
• Upload size
• Latency

Alerts

Recommended alerts:

1. Archival Failures

   rate(HistoryArchiver_errors[5m]) > 0.01
   

2. High Archival Latency

   histogram_quantile(0.99, rate(HistoryArchiver_latency_bucket[5m])) > 10000
   

3. Queue Backlog

   ArchivalQueueProcessor_pending_tasks > 10000
   

Archival Best Practices

1. Separate Archive URIs per Namespace

# Good
s3://temporal-archive/history/production
s3://temporal-archive/history/staging

# Avoid
s3://temporal-archive/history  # All namespaces

2. Configure Retention Appropriately

# Short retention, rely on archive
tctl namespace register production --retention 7

# Longer retention for critical workflows
tctl namespace register critical --retention 30

3. Test Archive Reads Regularly

# Verify archived data is accessible
tctl workflow show \
  --workflow-id archived-workflow \
  --namespace production

4. Monitor Storage Costs

• Use lifecycle policies to transition to cheaper storage
• Archive only necessary namespaces
• Set appropriate retention periods

5. Implement Archive Backup

Backup archive storage separately:

# S3 cross-region replication
aws s3api put-bucket-replication \
  --bucket temporal-archive-prod \
  --replication-configuration file://replication.json

Troubleshooting

Archival Failures

Symptoms:

• HistoryArchiver_errors metrics increasing
• Workflows not archived after retention

Common Causes:

1. Permissions - Missing S3/GCS write permissions
2. URI Invalid - Incorrect bucket name or path
3. Network - Cannot reach storage endpoint
4. Quota - Storage quota exceeded

Solutions:

# Check worker logs
kubectl logs -l app=temporal-worker

# Verify permissions
aws s3 ls s3://temporal-archive-prod/
gsutil ls gs://temporal-archive-prod/

# Test write access
echo "test" | aws s3 cp - s3://temporal-archive-prod/test.txt

Cannot Read Archived Data

Symptoms:

• workflow not found when querying old workflows
• Archive reads fail

Solutions:

1. Verify enableRead: true in global config
2. Check archive URI matches namespace config
3. Verify read permissions on storage
4. Check worker can access storage endpoint

High Archival Latency

Symptoms:

• Slow archival task processing
• Large queue backlog

Solutions:

1. Scale worker service horizontally
2. Increase worker task processing concurrency
3. Use faster storage tier
4. Optimize network path to storage

Advanced Configuration

Custom Archiver Implementation

Implement custom archiver for proprietary storage:

type HistoryArchiver interface {
    Archive(context.Context, URI, *ArchiveHistoryRequest, ...ArchiveOption) error
    Get(context.Context, URI, *GetHistoryRequest) (*GetHistoryResponse, error)
    ValidateURI(URI) error
}

type VisibilityArchiver interface {
    Archive(context.Context, URI, *ArchiveVisibilityRequest, ...ArchiveOption) error
    Query(context.Context, URI, *QueryVisibilityRequest) (*QueryVisibilityResponse, error)
    ValidateURI(URI) error
}

Register in provider:

import "go.temporal.io/server/common/archiver/provider"

func init() {
    provider.RegisterHistoryArchiver("custom", NewCustomHistoryArchiver)
    provider.RegisterVisibilityArchiver("custom", NewCustomVisibilityArchiver)
}

Archival Workflow Customization

Configure archival workflow via dynamic config:

worker.archiverConcurrency:
  - value: 50  # Parallel archival operations
    constraints: {}

worker.archiverMaxPendingArchivalTasks:
  - value: 10000
    constraints: {}

See Also

• Persistence - Primary data store configuration
• Monitoring - Archival metrics
• DLQ Management - Handling failed archival tasks