Chain Pattern: Building Multi-Stage Skill Workflows

When a single transformation is not enough, the chain pattern comes into play. This pattern connects multiple processing stages in sequence, where each stage builds on the output of the previous one. The result is greater than what any single stage could achieve alone.

The chain pattern is essential for research pipelines, document processing, code analysis, and any task that requires progressive refinement or multi-step transformation. In this guide, we will explore how to design, implement, and optimize chain-based skills for real-world applications.

Understanding the Chain Pattern

The chain pattern follows a sequential flow:

Input → [Stage 1] → [Stage 2] → [Stage 3] → Output

Each stage:

• Receives input from the previous stage
• Performs a specific transformation
• Passes its output to the next stage

The final output incorporates the work of all stages.

Core Characteristics

Chain-based skills have distinct properties:

Sequential Execution: Stages run in order. Stage 2 cannot start until Stage 1 completes.

Data Accumulation: Later stages can access outputs from all previous stages, not just the immediate predecessor.

Progressive Refinement: Each stage adds value, building toward the final result.

Clear Checkpoints: The boundary between stages provides natural points for validation, logging, and error handling.

Modular Design: Individual stages can be tested, replaced, or reused independently.

When to Use Chain

The chain pattern excels at:

• Research pipelines: Gather → Analyze → Synthesize → Report
• Document processing: Parse → Extract → Transform → Format
• Code analysis: Read → Parse → Analyze → Report
• Data transformation: Ingest → Clean → Transform → Validate → Output
• Content creation: Research → Outline → Draft → Edit → Polish

When to Avoid Chain

Chain is not ideal when:

• Stages do not have natural dependencies
• Processing could be parallelized for speed
• You need iterative refinement based on output quality
• The number or type of stages varies based on input

Consider Fan-Out/Fan-In for parallel processing, or Feedback Loop for iterative refinement.

Designing Chain Stages

Effective chains start with well-designed stages.

Stage Interface Design

Each stage should have a consistent interface:

## Stage Interface

### Input
```typescript
interface StageInput {
  // Data from previous stage
  previousOutput: any;

  // Accumulated context from all stages
  context: {
    original: any;        // Original input
    stage1Result?: any;   // Stage 1 output
    stage2Result?: any;   // Stage 2 output
    // ...
  };

  // Stage-specific configuration
  config: StageConfig;
}

Output

interface StageOutput {
  // This stage's result
  result: any;

  // Metadata about this stage
  metadata: {
    stageName: string;
    duration: number;
    success: boolean;
    notes?: string[];
  };

  // Pass-through for next stage
  context: StageContext;
}

### Stage Responsibilities

Define what each stage does and does not do:

```markdown
## Stage Design Principles

Each stage should:
1. Have a single, clear purpose
2. Accept input in a defined format
3. Produce output in a defined format
4. Handle its own errors gracefully
5. Be testable in isolation
6. Be replaceable without affecting other stages

Each stage should NOT:
1. Depend on stage implementation details
2. Make assumptions about future stages
3. Modify shared state unexpectedly
4. Skip error handling assuming later stages will catch it

Inter-Stage Communication

Design how data flows between stages:

## Data Flow Options

### Option 1: Pass Everything Forward
Each stage receives and forwards all previous data.
+ Complete information available
- Later stages may be overwhelmed

### Option 2: Pass Only What's Needed
Each stage receives only relevant previous outputs.
+ Cleaner, focused processing
- Requires careful interface design

### Option 3: Shared Context Object
Stages read/write to shared context.
+ Flexible access
- Risk of unintended mutations

Recommended: Option 2 with access to shared context for exceptional cases.

Implementing Chain Skills

Let us build a complete chain-based skill step by step.

Example: Research Pipeline

---
name: research-pipeline
description: Multi-stage research and synthesis pipeline
version: 1.0.0
---

# Research Pipeline

A four-stage chain for comprehensive research on any topic.

## Overview

Topic → [Gather] → [Analyze] → [Synthesize] → [Report] → Output

Each stage builds on previous work to create a comprehensive research report.

---

## Stage 1: Gather

Collect relevant information from available sources.

### Input
- `topic`: The research topic
- `scope`: "narrow" | "broad" | "exhaustive"
- `sources`: Array of source types to query

### Processing
1. Break topic into searchable queries
2. Search each enabled source
3. Collect relevant snippets
4. Deduplicate similar content
5. Rank by relevance

### Output
```json
{
  "sources": [
    {
      "title": "Source title",
      "url": "optional url",
      "content": "Relevant excerpt",
      "relevance": 0.95,
      "type": "documentation|article|code"
    }
  ],
  "queries": ["query 1", "query 2"],
  "coverage": {
    "topicsFound": ["topic1", "topic2"],
    "gaps": ["missing area"]
  }
}

Stage Metadata

• Source count
• Search duration
• Coverage assessment

---

Stage 2: Analyze

Extract insights and identify patterns from gathered sources.

Input

• gatherOutput: Stage 1 results
• analysisDepth: "quick" | "standard" | "deep"
• focusAreas: Optional specific areas to emphasize

Processing

1. Read all gathered sources
2. Identify key themes
3. Extract factual claims
4. Note contradictions or debates
5. Map relationships between concepts
6. Identify knowledge gaps

Output

{
  "themes": [
    {
      "name": "Theme name",
      "description": "What this theme covers",
      "supportingEvidence": ["source1", "source2"],
      "confidence": 0.9
    }
  ],
  "claims": [
    {
      "statement": "Factual claim",
      "sources": ["source refs"],
      "verified": true
    }
  ],
  "relationships": [
    {
      "from": "concept A",
      "to": "concept B",
      "type": "causes|enables|conflicts"
    }
  ],
  "gaps": ["Areas needing more research"]
}

---

Stage 3: Synthesize

Combine analysis into coherent understanding.

Input

• gatherOutput: Stage 1 results
• analyzeOutput: Stage 2 results
• synthesisStyle: "objective" | "opinionated" | "comparative"

Processing

1. Review themes and relationships
2. Construct narrative structure
3. Resolve or highlight contradictions
4. Build logical argument flow
5. Identify main conclusions
6. Note limitations

Output

{
  "narrative": {
    "introduction": "Opening context",
    "mainPoints": [
      {
        "point": "Key insight",
        "support": "Evidence",
        "significance": "Why it matters"
      }
    ],
    "conclusions": ["Main takeaway"],
    "limitations": ["What we don't know"]
  },
  "structure": {
    "sections": ["Section order"],
    "flow": "How sections connect"
  }
}

---

Stage 4: Report

Generate final formatted output.

Input

• All previous stage outputs
• format: "markdown" | "html" | "json"
• length: "summary" | "standard" | "comprehensive"
• audience: "technical" | "general" | "executive"

Processing

1. Select content based on length
2. Adjust language for audience
3. Format per specification
4. Add citations
5. Generate executive summary
6. Create table of contents

Output

{
  "report": "Formatted report content",
  "format": "markdown",
  "wordCount": 2500,
  "sections": ["toc entries"],
  "citations": [
    {
      "ref": "[1]",
      "source": "Source info"
    }
  ],
  "summary": "Executive summary paragraph"
}

---

Chain Configuration

pipeline:
  name: research-pipeline
  stages:
    - name: gather
      timeout: 60s
      retries: 2

    - name: analyze
      timeout: 90s
      retries: 1
      requires: [gather]

    - name: synthesize
      timeout: 60s
      requires: [gather, analyze]

    - name: report
      timeout: 30s
      requires: [gather, analyze, synthesize]

  errorHandling:
    strategy: "stop-on-error"
    savePartialResults: true

## Chain Control Flow

Chains need careful control flow management.

### Sequential Execution

The basic flow executes stages in order:

```markdown
## Execution Flow

1. Initialize chain context with original input
2. For each stage in order:
   a. Prepare stage input from context
   b. Execute stage
   c. Validate stage output
   d. Add output to context
   e. Check for continuation conditions
3. Return final output with full context

Conditional Stages

Some chains need to skip or include stages based on conditions:

## Conditional Execution

Stages may be conditional based on:

1. **Input conditions**
   ```yaml
   - name: deep-analysis
     condition: "input.depth === 'deep'"

1. Previous stage output

   - name: conflict-resolution
     condition: "analyze.hasConflicts === true"
   

2. Configuration flags

   - name: formatting
     condition: "config.generateReport === true"
   

When a stage is skipped:

• Log that it was skipped and why
• Pass previous output to next stage unchanged
• Note in metadata which stages were skipped

### Error Handling

Robust chains handle errors at each stage:

```markdown
## Error Handling Strategy

### Stage-Level Errors

When a stage fails:
1. Capture the error with full context
2. Check if error is recoverable
3. If recoverable, attempt retry with backoff
4. If not recoverable, decide chain behavior:
   - Stop: Halt chain, return partial results
   - Skip: Continue without this stage's output
   - Default: Use default/cached values

### Configuration
```yaml
errorHandling:
  gather:
    retries: 3
    retryDelay: 1000
    onFail: "stop"

  analyze:
    retries: 1
    onFail: "skip"
    skipMessage: "Analysis unavailable, continuing with raw data"

  report:
    retries: 0
    onFail: "default"
    defaultValue: "Report generation failed"

Error Response

{
  "success": false,
  "failedStage": "analyze",
  "error": {
    "type": "ProcessingError",
    "message": "Failed to extract themes",
    "stage": "analyze"
  },
  "partialResults": {
    "gather": { ... successful output ... }
  },
  "resumeToken": "token-for-retry"
}

## Optimizing Chain Performance

Chains can be slow if not optimized. Here are strategies to improve performance.

### Minimize Token Usage

Each stage consumes tokens. Be efficient:

```markdown
## Token Optimization

### Between Stages
- Pass only necessary data forward
- Summarize large outputs before passing
- Use references instead of duplicating data

### Within Stages
- Focus prompts on specific tasks
- Avoid repeating context unnecessarily
- Use structured output to reduce verbosity

### Example: Efficient Handoff
Instead of passing:
```json
{
  "sources": [... 50 full documents ...]
}

Pass:

{
  "sourceCount": 50,
  "sourceSummary": "Key points from sources",
  "sourceRefs": ["ref1", "ref2", ...],
  "fullSourcesAvailable": true
}

Later stages can request full sources if needed.

### Early Exit

Stop the chain early when possible:

```markdown
## Early Exit Conditions

Check after each stage:
- Goal already achieved?
- Insufficient data to continue?
- Error that prevents meaningful continuation?

```yaml
earlyExit:
  - after: gather
    if: "sourceCount === 0"
    response: "No sources found for topic"

  - after: analyze
    if: "themes.length === 0"
    response: "Unable to identify themes in sources"

  - after: synthesize
    if: "conclusions.length === 0"
    response: "Cannot draw conclusions from analysis"

### Caching Stage Results

Cache intermediate results for reuse:

```markdown
## Stage Caching

Cache stage outputs when:
- Input is identical to previous run
- Stage is expensive (time/tokens)
- Results don't depend on time-sensitive data

```yaml
caching:
  gather:
    enabled: true
    ttl: 3600
    keyFrom: ["topic", "scope", "sources"]

  analyze:
    enabled: true
    ttl: 1800
    keyFrom: ["gatherOutput.hash"]

  synthesize:
    enabled: false  # Depends on config

  report:
    enabled: false  # Quick, no need to cache

Cache invalidation:

• On explicit request
• On config change
• On TTL expiration
• On source update

## Testing Chain Skills

Chains require multi-level testing strategies.

### Unit Testing Stages

Test each stage in isolation:

```markdown
## Stage Unit Tests

### Gather Stage Tests
- Empty topic: Returns error
- Valid topic: Returns sources
- No results: Returns empty with appropriate message
- Source timeout: Retries, then graceful failure

### Analyze Stage Tests
- Empty sources: Returns "insufficient data"
- Single source: Extracts themes correctly
- Contradicting sources: Notes conflicts
- Large source set: Handles within timeout

### Synthesize Stage Tests
- Clear themes: Creates coherent narrative
- Conflicting data: Addresses contradictions
- Missing data: Notes limitations

### Report Stage Tests
- All formats: Generates valid output
- All lengths: Respects word count
- All audiences: Adjusts language appropriately

Integration Testing Chains

Test the full chain flow:

## Chain Integration Tests

### Happy Path
Input: Well-defined topic with good sources
Expected: Complete report with all sections

### Degraded Path
Input: Niche topic with few sources
Expected: Partial report with noted limitations

### Error Path
Input: Topic that will cause analysis failure
Expected: Graceful degradation, partial results

### Performance Test
Input: Complex topic requiring deep research
Expected: Completes within timeout, token budget

Snapshot Testing

Capture and compare outputs over time:

## Snapshot Tests

For each test case:
1. Run chain with fixed input
2. Compare output to stored snapshot
3. Flag differences for review

Benefits:
- Detect unexpected changes
- Track output quality over time
- Regression testing

Update snapshots when:
- Intentional improvements made
- Stage logic updated
- Output format changed

Real-World Chain Examples

Document Processing Pipeline

---
name: document-processor
description: Process documents through extraction, analysis, and indexing
---

# Document Processing Pipeline

## Stage 1: Ingest
- Accept document in various formats
- Normalize to plain text
- Extract metadata (author, date, etc.)

## Stage 2: Structure
- Identify document sections
- Extract headings and hierarchy
- Parse tables and lists

## Stage 3: Extract
- Pull key entities (names, dates, amounts)
- Identify important passages
- Tag content by topic

## Stage 4: Enrich
- Add external context
- Link to related documents
- Calculate relevance scores

## Stage 5: Index
- Generate searchable index
- Create summary
- Store structured output

Code Review Pipeline

---
name: code-review-pipeline
description: Multi-stage code review process
---

# Code Review Pipeline

## Stage 1: Parse
- Read source files
- Build AST
- Identify code structure

## Stage 2: Lint
- Apply style rules
- Check formatting
- Identify simple issues

## Stage 3: Analyze
- Complexity analysis
- Security scanning
- Performance checking

## Stage 4: Context
- Compare to codebase patterns
- Check against past issues
- Identify similar code

## Stage 5: Report
- Aggregate findings
- Prioritize issues
- Generate actionable report

Content Creation Pipeline

---
name: content-creator
description: Guided content creation workflow
---

# Content Creation Pipeline

## Stage 1: Research
- Understand the topic
- Gather reference material
- Identify angle and audience

## Stage 2: Outline
- Create structure
- Define key points
- Plan flow

## Stage 3: Draft
- Write initial content
- Follow outline
- Focus on completeness

## Stage 4: Edit
- Improve clarity
- Fix errors
- Strengthen arguments

## Stage 5: Polish
- Final formatting
- Add metadata
- Prepare for publication

Conclusion

The chain pattern transforms complex, multi-step processes into manageable, maintainable workflows. By breaking work into focused stages that build on each other, you create skills that are more powerful than monolithic alternatives.

Key principles for effective chain skills:

1. Clear stage boundaries: Each stage has defined input, output, and purpose
2. Progressive accumulation: Later stages access all previous context
3. Robust error handling: Failures are contained and handled gracefully
4. Performance awareness: Optimize token usage and consider caching
5. Comprehensive testing: Test stages individually and chains as a whole

Start with simple two or three stage chains. As you gain experience, build more sophisticated pipelines with conditional stages, parallel branches, and dynamic composition.

The chain pattern is your tool for tackling complex, multi-step challenges in a structured, reliable way.