> ## Documentation Index > Fetch the complete documentation index at: https://mintlify.com/temporalio/temporal/llms.txt > Use this file to discover all available pages before exploring further. # Chasm Architecture > Coordinated Heterogeneous Application State Machines framework # Chasm Architecture Chasm (Coordinated Heterogeneous Application State Machines) is Temporal's library for building complex, durable, distributed applications using a hierarchical state machine model. It provides a framework for implementing Temporal features that require sophisticated state management beyond traditional workflows. Chasm is an internal implementation framework, not a user-facing API. It is used to implement features like Scheduler, Worker Deployment, and other advanced Temporal capabilities. ## Motivation Temporal initially built all features using the core Workflow concept. However, certain features required: * **Complex state machines** with multiple concurrent components * **Long-lived executions** with evolving requirements * **Hierarchical composition** of state machines * **Sophisticated lifecycle management** * **Rich interaction patterns** beyond workflow signals/queries Chasm was created to provide a structured framework for building these features while maintaining Temporal's durability and consistency guarantees. ## Core Concepts ### Terminology Chasm introduces several key concepts: An **Archetype** is a type of state machine (e.g., Workflow, Activity, Scheduler). Each archetype defines: * State structure * Allowed transitions * Task types * Request-reply handlers An **Execution** is an instance of an archetype (e.g., a specific workflow execution, a specific scheduler). Each execution has: * A BusinessID (user-meaningful identifier) * Unique across non-terminal executions in a namespace * Can consist of multiple runs A **Component** is a node in the execution tree. An execution is structured as a tree where: * Root component represents the execution * Child components represent sub-state machines * Each component has its own state and tasks A **ComponentRef** uniquely identifies a component within an execution. Contains: * ExecutionKey (namespace + business ID) * ComponentPath (tree path to component) * Transition information (for multi-cluster failover) ```mermaid theme={null} flowchart TB subgraph Execution["Execution (e.g., Scheduler)"] Root["Root Component
(Scheduler)"] subgraph Children["Child Components"] Gen["Generator"] Inv["Invoker"] BF1["Backfiller 1"] BF2["Backfiller 2"] end Root --> Gen Root --> Inv Root --> BF1 Root --> BF2 end style Execution fill:#e1f5ff style Children fill:#fff4e1 ``` ## Architecture ### Hierarchical State Machines Chasm applications are structured as **trees of components**: * **Root component**: Represents the overall execution * **Child components**: Represent sub-state machines * **Leaf components**: Have no children Each component: * Has its own **state** (persisted) * Schedules its own **tasks** (timers, side effects) * Handles **requests** (via request-reply pattern) * Can create/delete child components dynamically ### State Management Component state is persisted using Temporal's existing infrastructure: * Stored in **Mutable State** (same as workflow state) * Uses **event sourcing** (state changes generate events) * **Atomic updates** via database transactions * **Cached** for performance ### Task Processing Components schedule tasks for execution: **Timer Tasks:** * Execute at a specific time * Drive time-based state transitions * Handled by History Service timer queue **Side Effect Tasks:** * Make external service calls * Execute immediately but with retries * Handled by History Service transfer queue ```mermaid theme={null} sequenceDiagram participant Comp as Component participant History as History Service participant Queue as Queue Processor participant External as External Service Comp->>History: Schedule TimerTask(10s) History->>History: Persist task Note over Queue: Wait 10 seconds Queue->>History: Process TimerTask History->>Comp: Execute task handler Comp->>History: Update state
Schedule SideEffectTask History->>History: Persist state & task Queue->>History: Process SideEffectTask History->>Comp: Execute side effect Comp->>External: RPC call External-->>Comp: Response Comp->>History: Update state ``` ## Component Lifecycle ### Component Creation Components are created dynamically: ```go theme={null} // Pseudo-code func (s *Scheduler) OnTriggerRequest(req TriggerRequest) { // Create a new Backfiller component backfiller := s.CreateChildComponent( "Backfiller", BackfillerState{ BackfillID: generateID(), Request: req, }, ) // Schedule initial task backfiller.ScheduleTask(BackfillerTask{ FireTime: time.Now(), }) } ``` ### Component Deletion Components can delete themselves: ```go theme={null} func (b *Backfiller) OnBackfillerTask() { if b.IsComplete() { b.DeleteSelf() return } // Continue processing b.ProcessNextBatch() } ``` ### State Transitions Components transition state via task handlers: ```go theme={null} func (c *Component) OnTimerTask() { // 1. Read current state state := c.GetState() // 2. Apply transition logic state.Counter++ state.LastUpdate = time.Now() // 3. Schedule next task if !state.IsComplete { c.ScheduleTask(TimerTask{ FireTime: time.Now().Add(time.Minute), }) } // 4. Persist state c.UpdateState(state) } ``` ## Request-Reply Pattern Chasm supports synchronous request-reply interactions: ### Request Types * **Queries**: Read-only, do not modify state * **Updates**: Modify state, wait for effect to be durable * **Signals**: Fire-and-forget state modifications ### Request Handling Components implement request handlers: ```go theme={null} type SchedulerComponent interface { // Query handler (read-only) OnDescribeRequest(req DescribeRequest) DescribeResponse // Update handler (modifies state) OnPauseRequest(req PauseRequest) PauseResponse // Signal handler (fire-and-forget) OnTriggerSignal(signal TriggerSignal) } ``` ### Routing Requests are routed to the appropriate component: * **Root requests**: Sent to root component * **Component-specific requests**: Include ComponentPath in request * **Broadcast requests**: Fan out to multiple components ## Example: Scheduler Implementation Temporal's Scheduler feature is implemented using Chasm: ### Component Structure ```mermaid theme={null} classDiagram class Scheduler { +Schedule schedule +Info info +OnDescribeRequest() +OnPauseRequest() +OnTriggerRequest() } class Generator { +Timestamp lastProcessedTime +OnGeneratorTask() } class Invoker { +BufferedStart[] bufferedStarts +OnProcessBufferTask() +OnExecuteTask() } class Backfiller { +UUID backfillID +BackfillRequest request +OnBackfillerTask() } Scheduler "1" --> "1" Generator : owns Scheduler "1" --> "1" Invoker : owns Scheduler "1" --> "0..*" Backfiller : owns ``` ### Task Flow **Generator Task:** * Fires periodically based on schedule spec * Buffers actions into Invoker * Reschedules itself for next interval **Invoker Tasks:** * `ProcessBufferTask`: Evaluates overlap policies * `ExecuteTask`: Starts workflows via RPC **Backfiller Task:** * Processes manual backfill requests * Buffers actions into Invoker * Deletes itself when complete ```mermaid theme={null} sequenceDiagram participant Gen as Generator participant Inv as Invoker participant BF as Backfiller participant Frontend Note over Gen: GeneratorTask fires Gen->>Inv: EnqueueBufferedStarts(actions) Inv->>Inv: Schedule ProcessBufferTask Gen->>Gen: Schedule next GeneratorTask Note over Inv: ProcessBufferTask fires Inv->>Inv: Resolve overlap policies Inv->>Inv: Mark eligible starts Inv->>Inv: Schedule ExecuteTask Note over Inv: ExecuteTask fires Inv->>Frontend: StartWorkflowExecution Frontend-->>Inv: Started Inv->>Inv: Record result Inv->>Inv: Schedule next ExecuteTask
(if retries pending) Note over BF: BackfillerTask fires BF->>Inv: EnqueueBufferedStarts(backfill actions) BF->>BF: Update lastProcessedTime BF->>BF: Schedule next BackfillerTask
or delete self if complete ``` See [Schedules Architecture](https://github.com/temporalio/temporal/blob/main/docs/architecture/schedules.md) for full details. ## Completion Callbacks Chasm supports **asynchronous completion callbacks** for side effects: ### Nexus Completion When a component starts a workflow: 1. Generate a **completion callback token** 2. Include token in `StartWorkflowExecution` request 3. When workflow completes, callback is triggered 4. Component receives notification and updates state ```go theme={null} // Start workflow with callback token := chasm.GenerateCallbackToken(componentRef) resp, err := client.StartWorkflowExecution(ctx, &workflowservice.StartWorkflowExecutionRequest{ // ... other fields CompletionCallbacks: []*commonpb.Callback{ { Variant: &commonpb.Callback_Nexus_{ Nexus: &commonpb.Callback_Nexus{ Url: callbackURL, Token: token, }, }, }, }, }) // Later, when workflow completes func (c *Component) OnCompletionCallback(result NexusResult) { c.state.LastResult = result c.UpdateState(c.state) } ``` ## Advantages of Chasm Build complex features by composing simpler components hierarchically. Components can be reused across different archetypes. Components can be tested independently. Components can be added/removed without breaking existing state. Only active components are loaded into memory. ## Comparison with Workflows | Aspect | Traditional Workflow | Chasm Execution | | ----------- | ------------------------- | ----------------------------------- | | Structure | Single state machine | Tree of state machines | | State | Workflow variables | Per-component state | | Tasks | Workflow & Activity Tasks | Component-specific tasks | | Composition | Child workflows | Component hierarchy | | Lifecycle | Start → Complete | Dynamic component creation/deletion | | Use Cases | Business workflows | Advanced Temporal features | ## Implementation Details ### Storage Chasm state is stored in the same tables as workflow state: * Component state in Mutable State columns * Component tasks in History task queues * Uses existing persistence infrastructure ### Code Organization Chasm library is located in `/chasm` directory: * `/chasm/lib`: Component implementations (Scheduler, Activity, NexusOperation) * `/chasm/interfaces.go`: Core interfaces * `/chasm/nexus_completion.go`: Completion callback support ```go theme={null} // Core Chasm interfaces type Component interface { // State management GetState() interface{} UpdateState(state interface{}) error // Task scheduling ScheduleTask(task Task) // Child management CreateChildComponent(componentType string, state interface{}) Component DeleteSelf() // Request handling HandleRequest(req Request) (Response, error) } ``` ### Integration with History Service Chasm components are executed by History Service: * Task execution goes through existing queue processors * State updates use existing transaction mechanisms * No separate service required ```go theme={null} // Code entrypoint: service/history/chasm_engine.go // ChasmEngine handles Chasm component execution ``` ## Use Cases Chasm is used to implement: Periodic workflow execution with backfill support Managing worker versioning and deployment Cross-namespace workflow invocations (Planned) Enhanced activity state management ## Development Guidelines ### When to Use Chasm Consider Chasm when: * Feature requires **multiple concurrent state machines** * Need **dynamic component creation/deletion** * Require **sophisticated lifecycle management** * State machine structure **evolves over time** ### When NOT to Use Chasm Don't use Chasm when: * Simple workflow is sufficient * State machine structure is fixed * No need for hierarchical composition * Complexity overhead not justified ### Best Practices Each component should have a single, well-defined responsibility. Store only essential state; derive everything else. Tasks may fail and be retried; ensure idempotency. Use diagrams to document component interactions and state transitions. ## Future Directions Chasm is evolving to support: * **More archetypes**: Activity, Query, Update as first-class Chasm components * **Better tooling**: Visualization and debugging tools * **Cross-cluster support**: Enhanced multi-cluster coordination * **Performance optimizations**: Lazy loading, state compression Chasm is an internal framework and its APIs may change. External developers should not depend on Chasm directly but use the stable public APIs that are built on top of it. ## Further Reading Detailed Scheduler implementation using Chasm How Chasm integrates with History Service State management foundation High-level system architecture