Insights
Founder and builder perspectives on AI tools, thinking patterns, and the new way of working
Showing 109-120 of 485
Agent-as-Judge: Letting an Agent Grade Another Agent's Work
Agent-as-judge gives evaluators tools, memory, and multi-step reasoning to grade full trajectories — but it inherits every bias a simple judge has, plus new ones.
Cheap Heuristics on 100% of Traffic, Expensive Judges on 10%
Running LLM-as-judge on every production request doesn't scale on cost or latency. Here's the sampling architecture teams actually run instead.
Building a Golden Dataset When You Don't Have Real Traffic Yet
You can't mine production logs for eval cases before you have users. Here's how solo builders and small teams bootstrap a golden dataset from zero.
Trajectory Evals: Grading the Path, Not Just the Answer
An agent that stumbles into the right answer through wasted tool calls isn't working. Trajectory evals grade the reasoning path, not just the output.
The Biases Baked Into Every AI Judge (and How to Cancel Them Out)
Position bias, verbosity bias, and self-preference bias distort every LLM-as-judge score. Here's what each looks like and the concrete fixes that work.
LLM-as-Judge Was the Start, Not the Finish
LLM-as-judge solved the eval bottleneck in 2024. By 2026 it's table stakes — here's what production teams layer on top of it now.
How Many Agents Is Too Many? A Cost/Latency Framework
Fleet size isn't a vibe — it's a function of decomposability, token budget, and how much compounding failure risk you can tolerate.
Debugging a Multi-Agent System When Something Goes Wrong
In a single-agent system, the bug is in the code. In a five-agent pipeline, the bug is in the seam — and most teams don't know how to look there.
The Supervisor Pattern: Claude Code's Default for a Reason
Developer tools converged on lead-agent-with-subagents before enterprise AI did. That order of adoption is a clue about why the pattern works.
Human-in-the-Loop Checkpoints for Autonomous Agent Fleets
Every approval gate you add slows a fleet down. Every gate you skip is a bet that nothing downstream needed a human. Where to place them.
Designing Handoffs Between Specialist Agents
The weakest point in a multi-agent pipeline is never the agents. It's the seam where context passes from one to the next — or doesn't.
Why 40% of Multi-Agent Pilots Failed in Their First Six Months
The models were capable enough. A postmortem look at why 40% of multi-agent pilots still didn't survive production — and it wasn't the AI's fault.