Building a Golden Dataset When You Don't Have Real Traffic Yet

Most writing about eval datasets has a quiet assumption baked into it: that you're sitting on months of production traffic, and the job is mining that traffic for representative cases, ambiguous edge cases, and known failures. That's genuinely good advice — for a team with production traffic. If you're a solo builder or a two-person team shipping the first version of an agent-powered skill or tool, that advice is close to useless, because the dataset you're supposed to mine doesn't exist yet. You need eval coverage before you have the users who would have generated it.

This is the exact position most builders on a marketplace like this one are in: shipping a skill, a small agent, or a tool with real functionality, but with a user base measured in dozens, not thousands. The golden dataset problem doesn't go away just because you're early — if anything it matters more, because you have less slack to absorb an agent that quietly does the wrong thing in front of one of your first twelve users. Here's how to build eval coverage without real traffic to mine.

Start from the failure modes you can already name

Before writing a single test case, sit down and list every way your agent could plausibly go wrong, based purely on what it does and what tools it has access to — not on anything you've observed yet, because you haven't observed anything yet. If it's a customer-support agent with a refund tool, the list includes: user asks for a refund outside policy, user asks for a refund with ambiguous order details, user tries to get the agent to bypass a stated limit through persistence, user asks something the agent has no tool for and needs to say so cleanly instead of hallucinating an answer.

This exercise produces more cases than it feels like it should, because most builders already know, intuitively, where their agent is fragile — they just haven't written it down as a test case. Treat this list as your seed set, and don't skip it in favor of jumping straight to synthetic generation, because a seed set grounded in your specific tool's real failure modes is worth more than a hundred synthetic cases generated without that grounding.

Use the model itself to generate adversarial variations

Once you have a seed set of ten or fifteen cases you know matter, use an LLM to generate variations on each one — same underlying scenario, different phrasing, different levels of politeness, different amounts of missing information, different attempts to push the agent past its stated boundaries. This is a legitimate and widely used technique precisely because the hard part — knowing which scenarios matter — was done by you in the seed set. The model is just amplifying coverage of situations you already identified as worth testing, not inventing the test strategy from scratch.

Be deliberate about generating adversarial cases specifically, not just paraphrases. A user who politely asks for something out of policy is a different test than a user who tries three different framings to get the same out-of-policy outcome. Agents that pass the polite version and fail the persistent version are agents with a real vulnerability that a small variation set built only from paraphrasing would never surface.

Borrow structure from adjacent domains, not from your own guesses

If you're building a scheduling agent and you've never run one in production, you're not actually starting from zero — scheduling as a task has well-understood edge cases that show up regardless of implementation: timezone ambiguity, double-booking, cancellation-and-rebooking races, requests that span a daylight-saving transition. These aren't specific to your agent; they're specific to the task category, and they're knowable in advance from domain knowledge rather than from your own traffic.

The same logic applies to almost any task category — payments, content moderation, code generation, data extraction. Each has a body of known-hard cases that exist independent of any specific implementation. Pulling from that body gets you closer to real-world coverage than pure introspection about your own product, because it captures failure modes you wouldn't have thought to invent yourself.

Recruit five real people before you recruit five hundred

Synthetic and self-generated cases get you real coverage, but they share a blind spot: they're generated by people who already know how the agent is supposed to be used. The single highest-leverage thing an early-stage builder can do for their eval set is put the actual agent in front of five to ten people who have never seen it and watch, without intervening, what they try to do with it. Real users phrase things in ways builders don't anticipate, combine requests in ways the seed-case exercise never considered, and hit the agent with the kind of ordinary confusion that only shows up when someone hasn't read the docs.

Every one of those sessions becomes eval cases — the mundane successful ones as regression coverage, the confused or failed ones as the highest-value cases in your entire set, because they're real evidence of a real gap rather than a hypothesis about one. This handful of real sessions is worth disproportionately more than a much larger batch of synthetic cases, because it's the only source of failure modes nobody on the team thought to imagine.

Keep the dataset small enough to actually look at

A common mistake in the opposite direction — once a team gets comfortable generating synthetic cases — is producing an eval set so large that no human ever reads through it end to end again. A golden dataset that's genuinely golden is not just fed to an automated judge and forgotten; someone periodically reads through the actual cases and actual outputs, which is also how you catch a judge that's drifted or a rubric that's stopped matching reality. For an early-stage agent, thirty to eighty well-chosen cases, reviewed by a human on a regular cadence, produce more reliable signal than a thousand cases nobody has read since the day they were generated.

This connects directly to the validation discipline covered later in this series, in "What Correlating Judge Scores to Human Ratings Actually Looks Like": a golden dataset is also the substrate you use to check whether your judge agrees with a human, and that check only works if a human is actually willing and able to read through the set.

Treat the dataset as a living artifact, not a one-time deliverable

The natural failure mode with a bootstrapped golden dataset is treating it as done the day it's built. It isn't. Every real production incident, every case a user hits that the dataset didn't cover, every time the agent's tools change — each of these is a signal to add a new case, not a one-off bug to patch and move past. The dataset that started as thirty seed cases from a founder's own intuition, amplified by adversarial generation and grounded by a handful of real user sessions, should look meaningfully different a year later, having absorbed every real failure the agent has actually produced in production.

Building eval coverage before you have traffic isn't a workaround you graduate out of once you have real users — it's the first stage of a dataset that keeps growing for as long as the agent is in production. The teams that treat it that way end up with eval sets that actually predict production behavior. The teams that treat the bootstrap phase as a placeholder to be replaced later usually never get around to the replacement, and keep shipping against a dataset that stopped reflecting reality months earlier.

Part of the "Evals for Agents" series on aiskill.market.