HTML to Markdown: The AI Pipeline

Every AI tool that works with web content faces the same problem: HTML is designed for browsers, not for language models. A typical documentation page is 15KB of HTML, but only 3KB is actual content. The rest is navigation bars, sidebars, footers, scripts, stylesheets, and structural markup that a browser renders visually but an LLM processes as tokens.

Converting HTML to clean markdown before feeding it to an AI saves tokens, improves response quality, and reduces processing time. This guide covers building a complete pipeline for this conversion -- from fetching web pages to producing clean, structured markdown ready for AI consumption.

Key Takeaways

• HTML-to-markdown conversion reduces token consumption by 60-80% for typical documentation and blog content
• The pipeline has four stages: fetch, clean, convert, structure -- each stage removes a layer of noise
• Readability extraction (like Mozilla's Readability) removes navigation chrome before conversion, producing cleaner output
• Structured markdown with consistent heading hierarchy helps AI tools parse and reference specific sections
• Caching converted content locally avoids redundant processing and reduces latency for frequently accessed documentation

The Pipeline Architecture

┌─────────┐     ┌─────────┐     ┌─────────┐     ┌──────────┐
│  Fetch   │────▶│  Clean   │────▶│ Convert  │────▶│ Structure │
│  (HTML)  │     │ (Extract)│     │   (MD)   │     │  (Index)  │
└─────────┘     └─────────┘     └─────────┘     └──────────┘

Fetch: Retrieve the raw HTML from a URL or local file. Clean: Extract the main content, removing navigation, ads, and chrome. Convert: Transform clean HTML to well-structured markdown. Structure: Add metadata, normalize headings, and index for retrieval.

Stage 1: Fetching HTML

The fetch stage retrieves raw HTML. For static pages, a simple HTTP request works. For JavaScript-rendered pages, you need a headless browser.

// lib/pipeline/fetch.ts

interface FetchResult {
  html: string
  url: string
  fetchedAt: string
  contentType: string
}

async function fetchPage(url: string): Promise<FetchResult> {
  const response = await fetch(url, {
    headers: {
      'User-Agent': 'ContentPipeline/1.0 (markdown conversion)',
      'Accept': 'text/html',
    },
  })
  
  if (!response.ok) {
    throw new Error(`Fetch failed: ${response.status} ${response.statusText}`)
  }
  
  return {
    html: await response.text(),
    url,
    fetchedAt: new Date().toISOString(),
    contentType: response.headers.get('content-type') || 'text/html',
  }
}

For JavaScript-rendered pages, use Playwright:

async function fetchRenderedPage(url: string): Promise<FetchResult> {
  const browser = await chromium.launch({ headless: true })
  const page = await browser.newPage()
  
  await page.goto(url, { waitUntil: 'networkidle' })
  const html = await page.content()
  
  await browser.close()
  
  return {
    html,
    url,
    fetchedAt: new Date().toISOString(),
    contentType: 'text/html',
  }
}

Stage 2: Cleaning HTML

This is the most important stage. Raw HTML contains massive amounts of noise. A typical page includes:

• Navigation bars and menus
• Sidebars with related links
• Cookie consent banners
• Footer with legal text
• Script tags with JavaScript
• Style tags with CSS
• Ads and tracking pixels
• Social sharing buttons

Mozilla's Readability library (used by Firefox's Reader View) does an excellent job of extracting the main content from a page.

// lib/pipeline/clean.ts
import { Readability } from '@mozilla/readability'
import { JSDOM } from 'jsdom'

interface CleanResult {
  title: string
  content: string  // Clean HTML of main content
  excerpt: string
  byline: string | null
  siteName: string | null
  wordCount: number
}

function cleanHTML(html: string, url: string): CleanResult {
  const dom = new JSDOM(html, { url })
  const reader = new Readability(dom.window.document)
  const article = reader.parse()
  
  if (!article) {
    throw new Error('Could not extract article content')
  }
  
  return {
    title: article.title,
    content: article.content,
    excerpt: article.excerpt || '',
    byline: article.byline,
    siteName: article.siteName,
    wordCount: article.textContent.split(/\s+/).length,
  }
}

Additional Cleaning

Readability handles most cases, but some content needs additional processing:

function additionalCleaning(html: string): string {
  const dom = new JSDOM(html)
  const doc = dom.window.document
  
  // Remove empty elements
  doc.querySelectorAll('p, div, span').forEach(el => {
    if (!el.textContent?.trim()) el.remove()
  })
  
  // Remove tracking attributes
  doc.querySelectorAll('[data-analytics], [data-track]').forEach(el => {
    el.removeAttribute('data-analytics')
    el.removeAttribute('data-track')
  })
  
  // Simplify image elements
  doc.querySelectorAll('img').forEach(img => {
    const src = img.getAttribute('src')
    const alt = img.getAttribute('alt')
    // Remove srcset, data attributes, loading attributes
    Array.from(img.attributes).forEach(attr => {
      if (!['src', 'alt'].includes(attr.name)) {
        img.removeAttribute(attr.name)
      }
    })
  })
  
  return doc.body.innerHTML
}

Stage 3: Converting to Markdown

With clean HTML in hand, convert it to well-structured markdown. The Turndown library handles this conversion.

// lib/pipeline/convert.ts
import TurndownService from 'turndown'

function createConverter(): TurndownService {
  const turndown = new TurndownService({
    headingStyle: 'atx',        // # style headings
    codeBlockStyle: 'fenced',   // ``` style code blocks
    bulletListMarker: '-',      // - for unordered lists
    emDelimiter: '*',           // *italic*
    strongDelimiter: '**',      // **bold**
  })
  
  // Custom rule: convert highlighted code blocks
  turndown.addRule('codeBlocks', {
    filter: (node) => {
      return node.nodeName === 'PRE' && 
             node.querySelector('code') !== null
    },
    replacement: (content, node) => {
      const code = node.querySelector('code')
      const language = detectLanguage(code)
      const text = code.textContent || ''
      return `\n\`\`\`${language}\n${text.trim()}\n\`\`\`\n`
    },
  })
  
  // Custom rule: handle tables
  turndown.addRule('tables', {
    filter: 'table',
    replacement: (content, node) => {
      return convertTableToMarkdown(node)
    },
  })
  
  return turndown
}

function convertToMarkdown(cleanHtml: string): string {
  const converter = createConverter()
  return converter.turndown(cleanHtml)
}

function detectLanguage(codeElement: Element | null): string {
  if (!codeElement) return ''
  const classes = codeElement.className
  const match = classes.match(/language-(\w+)/)
  return match ? match[1] : ''
}

Stage 4: Structuring the Output

The final stage adds metadata and normalizes the markdown for consistent AI consumption.

// lib/pipeline/structure.ts

interface StructuredContent {
  frontmatter: {
    title: string
    source: string
    fetchedAt: string
    wordCount: number
    sections: string[]
  }
  markdown: string
}

function structureContent(
  title: string,
  markdown: string,
  url: string,
  wordCount: number
): StructuredContent {
  // Normalize headings (ensure h1 is title, content starts at h2)
  const normalizedMd = normalizeHeadings(markdown, title)
  
  // Extract section titles for indexing
  const sections = extractSections(normalizedMd)
  
  return {
    frontmatter: {
      title,
      source: url,
      fetchedAt: new Date().toISOString(),
      wordCount,
      sections,
    },
    markdown: normalizedMd,
  }
}

function normalizeHeadings(markdown: string, title: string): string {
  // Ensure consistent heading hierarchy
  const lines = markdown.split('\n')
  let minLevel = 6
  
  for (const line of lines) {
    const match = line.match(/^(#{1,6}) /)
    if (match) {
      minLevel = Math.min(minLevel, match[1].length)
    }
  }
  
  if (minLevel === 1) {
    // Bump all headings down by one level
    return lines.map(line => {
      return line.replace(/^(#{1,5}) /, (_, hashes) => '#' + hashes + ' ')
    }).join('\n')
  }
  
  return markdown
}

function extractSections(markdown: string): string[] {
  return markdown
    .split('\n')
    .filter(line => line.startsWith('## '))
    .map(line => line.replace('## ', ''))
}

The Complete Pipeline

// lib/pipeline/index.ts

async function processUrl(url: string): Promise<StructuredContent> {
  // Stage 1: Fetch
  const fetched = await fetchPage(url)
  
  // Stage 2: Clean
  const cleaned = cleanHTML(fetched.html, url)
  
  // Stage 3: Convert
  const markdown = convertToMarkdown(cleaned.content)
  
  // Stage 4: Structure
  const structured = structureContent(
    cleaned.title,
    markdown,
    url,
    cleaned.wordCount
  )
  
  return structured
}

// Usage
const content = await processUrl('https://docs.example.com/api-reference')
console.log(`Title: ${content.frontmatter.title}`)
console.log(`Words: ${content.frontmatter.wordCount}`)
console.log(`Sections: ${content.frontmatter.sections.join(', ')}`)
console.log(content.markdown)

Caching for Performance

Processing the same URL repeatedly wastes time and bandwidth. Cache converted content locally.

// lib/pipeline/cache.ts
import { readFile, writeFile, mkdir } from 'fs/promises'
import { createHash } from 'crypto'

const CACHE_DIR = '.content-cache'
const CACHE_TTL = 24 * 60 * 60 * 1000  // 24 hours

function getCacheKey(url: string): string {
  return createHash('md5').update(url).digest('hex')
}

async function getCached(url: string): Promise<StructuredContent | null> {
  const key = getCacheKey(url)
  const path = `${CACHE_DIR}/${key}.json`
  
  try {
    const data = JSON.parse(await readFile(path, 'utf-8'))
    const age = Date.now() - new Date(data.frontmatter.fetchedAt).getTime()
    
    if (age < CACHE_TTL) {
      return data
    }
  } catch {
    // Cache miss
  }
  
  return null
}

async function setCache(url: string, content: StructuredContent): Promise<void> {
  await mkdir(CACHE_DIR, { recursive: true })
  const key = getCacheKey(url)
  await writeFile(`${CACHE_DIR}/${key}.json`, JSON.stringify(content, null, 2))
}

Token Savings Analysis

Here are real measurements from processing common documentation pages.

For a 76% average reduction, every dollar you spend on AI tokens goes 4x further when processing pre-converted content.

This pipeline connects naturally to the llms.txt approach -- convert your documentation to markdown, then serve it as llms-full.txt for AI tools to consume directly.

FAQ

Does this work for all websites?

It works for content-heavy pages (documentation, blogs, articles). It does not work well for web applications, SPAs with dynamic content, or pages behind authentication. For those, use the headless browser approach.

Is web scraping legal?

Scraping publicly accessible web pages for personal use is generally legal. Scraping at scale, ignoring robots.txt, or republishing scraped content may violate terms of service or copyright law. Always respect robots.txt and rate limits.

How do I handle images?

The pipeline preserves image references as markdown image syntax (![alt](url)). For offline use, add a step that downloads images and rewrites URLs to local paths. For AI consumption, image alt text is often sufficient.

What about code examples in documentation?

The Turndown converter preserves code blocks with language detection. Code examples in <pre><code> blocks are converted to fenced markdown code blocks with the appropriate language tag.

Can I process an entire documentation site at once?

Yes. Crawl the sitemap, process each URL through the pipeline, and concatenate the results. Add a table of contents at the top. The result is a single markdown file containing an entire documentation site, suitable for loading into an AI's context window. See our CLI commands reference for how Claude Code handles large context files.

Explore production-ready AI skills at aiskill.market/browse or submit your own skill to the marketplace.

Sources

• Mozilla Readability - Content extraction library used by Firefox Reader View
• Turndown - HTML to markdown converter
• llms.txt Specification - Standard for AI-readable documentation
• Playwright Documentation - Headless browser for JavaScript-rendered pages