Chain Pattern: Building Multi-Stage Skill Workflows
Learn the chain pattern for AI skills that process data through sequential stages, building powerful pipelines for research, analysis, and transformation.
Chain Pattern: Building Multi-Stage Skill Workflows
When a single transformation is not enough, the chain pattern comes into play. This pattern connects multiple processing stages in sequence, where each stage builds on the output of the previous one. The result is greater than what any single stage could achieve alone.
The chain pattern is essential for research pipelines, document processing, code analysis, and any task that requires progressive refinement or multi-step transformation. In this guide, we will explore how to design, implement, and optimize chain-based skills for real-world applications.
Understanding the Chain Pattern
The chain pattern follows a sequential flow:
Input → [Stage 1] → [Stage 2] → [Stage 3] → Output
Each stage:
- Receives input from the previous stage
- Performs a specific transformation
- Passes its output to the next stage
The final output incorporates the work of all stages.
Core Characteristics
Chain-based skills have distinct properties:
Sequential Execution: Stages run in order. Stage 2 cannot start until Stage 1 completes.
Data Accumulation: Later stages can access outputs from all previous stages, not just the immediate predecessor.
Progressive Refinement: Each stage adds value, building toward the final result.
Clear Checkpoints: The boundary between stages provides natural points for validation, logging, and error handling.
Modular Design: Individual stages can be tested, replaced, or reused independently.
When to Use Chain
The chain pattern excels at:
- Research pipelines: Gather → Analyze → Synthesize → Report
- Document processing: Parse → Extract → Transform → Format
- Code analysis: Read → Parse → Analyze → Report
- Data transformation: Ingest → Clean → Transform → Validate → Output
- Content creation: Research → Outline → Draft → Edit → Polish
When to Avoid Chain
Chain is not ideal when:
- Stages do not have natural dependencies
- Processing could be parallelized for speed
- You need iterative refinement based on output quality
- The number or type of stages varies based on input
Consider Fan-Out/Fan-In for parallel processing, or Feedback Loop for iterative refinement.
Designing Chain Stages
Effective chains start with well-designed stages.
Stage Interface Design
Each stage should have a consistent interface:
## Stage Interface
### Input
```typescript
interface StageInput {
// Data from previous stage
previousOutput: any;
// Accumulated context from all stages
context: {
original: any; // Original input
stage1Result?: any; // Stage 1 output
stage2Result?: any; // Stage 2 output
// ...
};
// Stage-specific configuration
config: StageConfig;
}
Output
interface StageOutput {
// This stage's result
result: any;
// Metadata about this stage
metadata: {
stageName: string;
duration: number;
success: boolean;
notes?: string[];
};
// Pass-through for next stage
context: StageContext;
}
### Stage Responsibilities
Define what each stage does and does not do:
```markdown
## Stage Design Principles
Each stage should:
1. Have a single, clear purpose
2. Accept input in a defined format
3. Produce output in a defined format
4. Handle its own errors gracefully
5. Be testable in isolation
6. Be replaceable without affecting other stages
Each stage should NOT:
1. Depend on stage implementation details
2. Make assumptions about future stages
3. Modify shared state unexpectedly
4. Skip error handling assuming later stages will catch it
Inter-Stage Communication
Design how data flows between stages:
## Data Flow Options
### Option 1: Pass Everything Forward
Each stage receives and forwards all previous data.
+ Complete information available
- Later stages may be overwhelmed
### Option 2: Pass Only What's Needed
Each stage receives only relevant previous outputs.
+ Cleaner, focused processing
- Requires careful interface design
### Option 3: Shared Context Object
Stages read/write to shared context.
+ Flexible access
- Risk of unintended mutations
Recommended: Option 2 with access to shared context for exceptional cases.
Implementing Chain Skills
Let us build a complete chain-based skill step by step.
Example: Research Pipeline
---
name: research-pipeline
description: Multi-stage research and synthesis pipeline
version: 1.0.0
---
# Research Pipeline
A four-stage chain for comprehensive research on any topic.
## Overview
Topic → [Gather] → [Analyze] → [Synthesize] → [Report] → Output
Each stage builds on previous work to create a comprehensive research report.
---
## Stage 1: Gather
Collect relevant information from available sources.
### Input
- `topic`: The research topic
- `scope`: "narrow" | "broad" | "exhaustive"
- `sources`: Array of source types to query
### Processing
1. Break topic into searchable queries
2. Search each enabled source
3. Collect relevant snippets
4. Deduplicate similar content
5. Rank by relevance
### Output
```json
{
"sources": [
{
"title": "Source title",
"url": "optional url",
"content": "Relevant excerpt",
"relevance": 0.95,
"type": "documentation|article|code"
}
],
"queries": ["query 1", "query 2"],
"coverage": {
"topicsFound": ["topic1", "topic2"],
"gaps": ["missing area"]
}
}
Stage Metadata
- Source count
- Search duration
- Coverage assessment
Stage 2: Analyze
Extract insights and identify patterns from gathered sources.
Input
gatherOutput: Stage 1 resultsanalysisDepth: "quick" | "standard" | "deep"focusAreas: Optional specific areas to emphasize
Processing
- Read all gathered sources
- Identify key themes
- Extract factual claims
- Note contradictions or debates
- Map relationships between concepts
- Identify knowledge gaps
Output
{
"themes": [
{
"name": "Theme name",
"description": "What this theme covers",
"supportingEvidence": ["source1", "source2"],
"confidence": 0.9
}
],
"claims": [
{
"statement": "Factual claim",
"sources": ["source refs"],
"verified": true
}
],
"relationships": [
{
"from": "concept A",
"to": "concept B",
"type": "causes|enables|conflicts"
}
],
"gaps": ["Areas needing more research"]
}
Stage 3: Synthesize
Combine analysis into coherent understanding.
Input
gatherOutput: Stage 1 resultsanalyzeOutput: Stage 2 resultssynthesisStyle: "objective" | "opinionated" | "comparative"
Processing
- Review themes and relationships
- Construct narrative structure
- Resolve or highlight contradictions
- Build logical argument flow
- Identify main conclusions
- Note limitations
Output
{
"narrative": {
"introduction": "Opening context",
"mainPoints": [
{
"point": "Key insight",
"support": "Evidence",
"significance": "Why it matters"
}
],
"conclusions": ["Main takeaway"],
"limitations": ["What we don't know"]
},
"structure": {
"sections": ["Section order"],
"flow": "How sections connect"
}
}
Stage 4: Report
Generate final formatted output.
Input
- All previous stage outputs
format: "markdown" | "html" | "json"length: "summary" | "standard" | "comprehensive"audience: "technical" | "general" | "executive"
Processing
- Select content based on length
- Adjust language for audience
- Format per specification
- Add citations
- Generate executive summary
- Create table of contents
Output
{
"report": "Formatted report content",
"format": "markdown",
"wordCount": 2500,
"sections": ["toc entries"],
"citations": [
{
"ref": "[1]",
"source": "Source info"
}
],
"summary": "Executive summary paragraph"
}
Chain Configuration
pipeline:
name: research-pipeline
stages:
- name: gather
timeout: 60s
retries: 2
- name: analyze
timeout: 90s
retries: 1
requires: [gather]
- name: synthesize
timeout: 60s
requires: [gather, analyze]
- name: report
timeout: 30s
requires: [gather, analyze, synthesize]
errorHandling:
strategy: "stop-on-error"
savePartialResults: true
## Chain Control Flow
Chains need careful control flow management.
### Sequential Execution
The basic flow executes stages in order:
```markdown
## Execution Flow
1. Initialize chain context with original input
2. For each stage in order:
a. Prepare stage input from context
b. Execute stage
c. Validate stage output
d. Add output to context
e. Check for continuation conditions
3. Return final output with full context
Conditional Stages
Some chains need to skip or include stages based on conditions:
## Conditional Execution
Stages may be conditional based on:
1. **Input conditions**
```yaml
- name: deep-analysis
condition: "input.depth === 'deep'"
-
Previous stage output
- name: conflict-resolution condition: "analyze.hasConflicts === true" -
Configuration flags
- name: formatting condition: "config.generateReport === true"
When a stage is skipped:
- Log that it was skipped and why
- Pass previous output to next stage unchanged
- Note in metadata which stages were skipped
### Error Handling
Robust chains handle errors at each stage:
```markdown
## Error Handling Strategy
### Stage-Level Errors
When a stage fails:
1. Capture the error with full context
2. Check if error is recoverable
3. If recoverable, attempt retry with backoff
4. If not recoverable, decide chain behavior:
- Stop: Halt chain, return partial results
- Skip: Continue without this stage's output
- Default: Use default/cached values
### Configuration
```yaml
errorHandling:
gather:
retries: 3
retryDelay: 1000
onFail: "stop"
analyze:
retries: 1
onFail: "skip"
skipMessage: "Analysis unavailable, continuing with raw data"
report:
retries: 0
onFail: "default"
defaultValue: "Report generation failed"
Error Response
{
"success": false,
"failedStage": "analyze",
"error": {
"type": "ProcessingError",
"message": "Failed to extract themes",
"stage": "analyze"
},
"partialResults": {
"gather": { ... successful output ... }
},
"resumeToken": "token-for-retry"
}
## Optimizing Chain Performance
Chains can be slow if not optimized. Here are strategies to improve performance.
### Minimize Token Usage
Each stage consumes tokens. Be efficient:
```markdown
## Token Optimization
### Between Stages
- Pass only necessary data forward
- Summarize large outputs before passing
- Use references instead of duplicating data
### Within Stages
- Focus prompts on specific tasks
- Avoid repeating context unnecessarily
- Use structured output to reduce verbosity
### Example: Efficient Handoff
Instead of passing:
```json
{
"sources": [... 50 full documents ...]
}
Pass:
{
"sourceCount": 50,
"sourceSummary": "Key points from sources",
"sourceRefs": ["ref1", "ref2", ...],
"fullSourcesAvailable": true
}
Later stages can request full sources if needed.
### Early Exit
Stop the chain early when possible:
```markdown
## Early Exit Conditions
Check after each stage:
- Goal already achieved?
- Insufficient data to continue?
- Error that prevents meaningful continuation?
```yaml
earlyExit:
- after: gather
if: "sourceCount === 0"
response: "No sources found for topic"
- after: analyze
if: "themes.length === 0"
response: "Unable to identify themes in sources"
- after: synthesize
if: "conclusions.length === 0"
response: "Cannot draw conclusions from analysis"
### Caching Stage Results
Cache intermediate results for reuse:
```markdown
## Stage Caching
Cache stage outputs when:
- Input is identical to previous run
- Stage is expensive (time/tokens)
- Results don't depend on time-sensitive data
```yaml
caching:
gather:
enabled: true
ttl: 3600
keyFrom: ["topic", "scope", "sources"]
analyze:
enabled: true
ttl: 1800
keyFrom: ["gatherOutput.hash"]
synthesize:
enabled: false # Depends on config
report:
enabled: false # Quick, no need to cache
Cache invalidation:
- On explicit request
- On config change
- On TTL expiration
- On source update
## Testing Chain Skills
Chains require multi-level testing strategies.
### Unit Testing Stages
Test each stage in isolation:
```markdown
## Stage Unit Tests
### Gather Stage Tests
- Empty topic: Returns error
- Valid topic: Returns sources
- No results: Returns empty with appropriate message
- Source timeout: Retries, then graceful failure
### Analyze Stage Tests
- Empty sources: Returns "insufficient data"
- Single source: Extracts themes correctly
- Contradicting sources: Notes conflicts
- Large source set: Handles within timeout
### Synthesize Stage Tests
- Clear themes: Creates coherent narrative
- Conflicting data: Addresses contradictions
- Missing data: Notes limitations
### Report Stage Tests
- All formats: Generates valid output
- All lengths: Respects word count
- All audiences: Adjusts language appropriately
Integration Testing Chains
Test the full chain flow:
## Chain Integration Tests
### Happy Path
Input: Well-defined topic with good sources
Expected: Complete report with all sections
### Degraded Path
Input: Niche topic with few sources
Expected: Partial report with noted limitations
### Error Path
Input: Topic that will cause analysis failure
Expected: Graceful degradation, partial results
### Performance Test
Input: Complex topic requiring deep research
Expected: Completes within timeout, token budget
Snapshot Testing
Capture and compare outputs over time:
## Snapshot Tests
For each test case:
1. Run chain with fixed input
2. Compare output to stored snapshot
3. Flag differences for review
Benefits:
- Detect unexpected changes
- Track output quality over time
- Regression testing
Update snapshots when:
- Intentional improvements made
- Stage logic updated
- Output format changed
Real-World Chain Examples
Document Processing Pipeline
---
name: document-processor
description: Process documents through extraction, analysis, and indexing
---
# Document Processing Pipeline
## Stage 1: Ingest
- Accept document in various formats
- Normalize to plain text
- Extract metadata (author, date, etc.)
## Stage 2: Structure
- Identify document sections
- Extract headings and hierarchy
- Parse tables and lists
## Stage 3: Extract
- Pull key entities (names, dates, amounts)
- Identify important passages
- Tag content by topic
## Stage 4: Enrich
- Add external context
- Link to related documents
- Calculate relevance scores
## Stage 5: Index
- Generate searchable index
- Create summary
- Store structured output
Code Review Pipeline
---
name: code-review-pipeline
description: Multi-stage code review process
---
# Code Review Pipeline
## Stage 1: Parse
- Read source files
- Build AST
- Identify code structure
## Stage 2: Lint
- Apply style rules
- Check formatting
- Identify simple issues
## Stage 3: Analyze
- Complexity analysis
- Security scanning
- Performance checking
## Stage 4: Context
- Compare to codebase patterns
- Check against past issues
- Identify similar code
## Stage 5: Report
- Aggregate findings
- Prioritize issues
- Generate actionable report
Content Creation Pipeline
---
name: content-creator
description: Guided content creation workflow
---
# Content Creation Pipeline
## Stage 1: Research
- Understand the topic
- Gather reference material
- Identify angle and audience
## Stage 2: Outline
- Create structure
- Define key points
- Plan flow
## Stage 3: Draft
- Write initial content
- Follow outline
- Focus on completeness
## Stage 4: Edit
- Improve clarity
- Fix errors
- Strengthen arguments
## Stage 5: Polish
- Final formatting
- Add metadata
- Prepare for publication
Conclusion
The chain pattern transforms complex, multi-step processes into manageable, maintainable workflows. By breaking work into focused stages that build on each other, you create skills that are more powerful than monolithic alternatives.
Key principles for effective chain skills:
- Clear stage boundaries: Each stage has defined input, output, and purpose
- Progressive accumulation: Later stages access all previous context
- Robust error handling: Failures are contained and handled gracefully
- Performance awareness: Optimize token usage and consider caching
- Comprehensive testing: Test stages individually and chains as a whole
Start with simple two or three stage chains. As you gain experience, build more sophisticated pipelines with conditional stages, parallel branches, and dynamic composition.
The chain pattern is your tool for tackling complex, multi-step challenges in a structured, reliable way.