Promptfoo Evaluation

---

name: promptfoo-evaluation description: Configures and runs LLM evaluation using Promptfoo framework. Use when setting up prompt testing, creating evaluation configs (promptfooconfig.yaml), writing Python custom assertions, implementing llm-rubric for LLM-as-judge, or managing few-shot examples in prompts. Triggers on keywords like "promptfoo", "eval", "LLM evaluation", "prompt testing", or "model comparison".

Promptfoo Evaluation

Overview

This skill provides guidance for configuring and running LLM evaluations using Promptfoo, an open-source CLI tool for testing and comparing LLM outputs.

Quick Start

# Initialize a new evaluation project
npx promptfoo@latest init

# Run evaluation
npx promptfoo@latest eval

# View results in browser
npx promptfoo@latest view

Configuration Structure

A typical Promptfoo project structure:

project/
├── promptfooconfig.yaml    # Main configuration
├── prompts/
│   ├── system.md      

## Code Examples

### README.md

Skill Demonstrations

This directory contains automated demonstrations of each skill using VHS (Video Home System) by Charmbracelet.

VHS allows you to write terminal recordings as code, making demos reproducible, version-controllable, and typo-free.

📁 Demo Structure

demos/
├── skill-creator/
│   ├── init-skill.tape          # Initialize a new skill
│   ├── validate-skill.tape       # Validate skill quality
│   └── package-skill.tape        # Package for distribution
├── github-ops/
│   └── create-pr.tape            # Create pull requests
├── markdown-tools/
│   └── convert-docs.tape         # Convert documents
└── generate_all_demos.sh         # Generate all GIFs

🎬 Generating Demos

Prerequisites

Install VHS:

macOS:

brew install vhs

Linux (with Go):

go install github.com/charmbracelet/vhs@latest

Or download from: https://github.com/charmbracelet/vhs/releases

Generate All Demos

### generate_all_demos.sh

#!/bin/bash set -e

echo "================================================" echo "Generating Claude Code Skills Demo GIFs" echo "================================================" echo ""

Colors for output

GREEN='\033[0;32m' YELLOW='\033[1;33m' RED='\033[0;31m' NC='\033[0m' # No Color

Check if VHS is installed

if ! command -v vhs &> /dev/null; then echo -e "${RED}Error: VHS is not installed!${NC}" echo "" echo "Install VHS to generate demo GIFs:" echo "" echo "macOS:" echo " brew install vhs" echo "" echo "Linux (with Go):" echo " go install github.com/charmbracelet/vhs@latest" echo "" echo "Or download from: https://github.com/charmbracelet/vhs/releases" echo "" exit 1 fi

echo -e "${GREEN}✓ VHS found${NC}" echo ""

Find all .tape files

TAPE_FILES=$(find demos -name "*.tape" -type f | sort) TOTAL=$(echo "$TAPE_FILES" | wc -l | tr -d ' ') CURRENT=0

echo "Found $TOTAL demo tape files" echo ""

Generate each demo

for tape_fi

### index.html