SShogo Learn
← Library

Guide · 2026-10-28

Turn a real moment from your workspace into an eval

What you'll have running: a small suite of evals built from moments that actually happened in your workspace. Each one is a prompt, a seeded workspace, mocked tools, criteria scored in three phases, and a list of anti-patterns. You re-run the suite whenever you change your agent, and it tells you whether the behaviour still holds.

We run this. Our agent runtime holds about 378 evals in 76 suites, and every one has the five parts below. The five that came out of the game build are a single file, captured from production on 2026-10-16, with its moments named tag-me, fold-update, ack, stuck and restart. What this page gives you is the shape, not our repo: the eval files are ours and private, so the block below is a template to fill in rather than a copy of one of them.

What it is

An eval is a little description of a good teammate, written so you can replay it. Five parts, and nothing else:

prompt            the one message the agent read
seeded workspace  what was already there when it read it
mocked tools      what every tool returned that day
criteria          scored in three phases: intention · execution · interaction
anti-patterns     what fails the eval even if everything else is green
  • Prompt. The message, close to verbatim. Not a summary of the moment — the moment.
  • Seeded workspace. The cards, files and earlier messages the agent could see. This is what makes it a replay instead of a hypothetical.
  • Mocked tools. Every tool it could call, returning what it really returned. No provider, no network, nothing paid, so the eval is fast and repeatable.
  • Criteria in three phases. Intention: did it understand what I meant? Execution: did it do it, and does the thing actually run? Interaction: did it work with me well — no fake "done", no leaked key, no asking twice?
  • Anti-patterns. The behaviours that must fail the eval, named in advance.
a moment you had to explain twice
        │   paste the thread
        ▼
  1 PICK    can you replay it from what you logged?
  2 PROMPT  the message, verbatim
  3 SEED    the state it arrived in
  4 PHASES  intention · execution · interaction
  5 PASS    the outcome you grade, and what fails it
        │
        ▼
  one file you re-run on every change to the agent

Why it works

  • An eval is not a unit test. A unit test says: give it 2 and 2, expect 4, forever. An agent has a hundred good ways to build the same feature, so what you care about is how it behaves. That's why an eval describes a behaviour instead of an answer.
  • Grade the outcome, not the steps. Anthropic's evals team ran Claude on an airline-support task — change a customer's flight within the rules — and it scored as a fail. It had found a path through the policy that got the customer a better deal than the test expected. Grade the path and a smarter agent looks like a worse one. Grade the outcome and your evals improve as the agent does.
  • The moment is already written down. Every tool call and every turn's health is logged, so the eval is a query away rather than a memory exercise. Our prod-baseline.sql keeps one query per failure signature, tagged so an eval ID maps 1:1 to a production number. First time round that loop, malformed file reads went from 22% to 0%.
  • One case is half a behaviour. "It should act" and "it should ask" are the same behaviour seen twice. Write both or you've taught it to never ask.
  • It holds the behaviour still while you change everything else. Prompt, model, tool list, agent instructions: re-run the suite, read the score.

Step 1: pick the moment

The rule: a moment you never want to explain twice. If you've typed the same correction to your agent more than once, that's the one.

The filter: can you replay it? You need the message, the state it arrived in, and what the tools returned. If you can't find those three, it isn't ready yet — go and log them first.

From the game project I'd been building with three agents, the five that made it were all small decisions with two defensible answers:

tag-me        I asked to be tagged when cards finish. Does the lead act now, or wait for its hourly check?
fold-update   an engineer finished a card. Review it now, or wait to be asked?
ack           an engineer replied "got it, saved". Does that need a reply at all?
stuck         a card came back twice with the same fix. Hand it back a third time, or ask me?
restart       I restarted everything. What is supposed to be in flight afterwards?

Yours are in the same places: a card thread where you had to step in, a reply you typed twice, a tool call that failed the same way all week, or a word your users keep sending — we found 846 bare "continue" messages in five days, and that became an eval about finishing instead of asking.

Step 2: write the prompt

  • Verbatim, trimmed. Cut the greetings and the stage directions; keep the words that carried the request.
  • One message. If the real moment was three turns, put the first two in the seeded history and use only the third as the prompt.
  • No hints. Don't write "and don't ask me first" into the prompt. That's a criterion, not an input.
  • Keep it as vague as it really was. If you were vague that day, be vague here — that's the part being tested.

Step 3: seed the workspace

Write down what was true the second before the agent replied:

cards        key, title, column, assignee  (the real board, not a tidy one)
files        the paths that existed, with the content that mattered
messages     earlier turns the agent could see, in order
connections  a fake id per integration: gh-test, jira-test, none
clock        the time and day, if a schedule was involved

Two rules. Seed the mess: if two cards were half-done and one was stale, seed that — a tidy workspace tests a day that never happened. Never seed a secret: fake ids and dummy tokens only, or your eval is the leak.

Step 4: name the three phases

Under each phase, write checks you could hand to someone else to verify:

  • Intention — did it understand what I meant? Moved GAME-12 to Review, not Done. Touched no other card. Didn't rename anything.
  • Execution — did it do it, and does it run? The app boots. The route returns the record. The menu opens from the HUD. This is the phase where you boot the thing and hit it; a green build is not evidence.
  • Interaction — did it work with me well? One message on the card that tags me. No "done" before the checks ran. Nothing pasted back that I sent it in confidence. No question I'd already answered.

If a phase has no check, you didn't grade it. The commonest eval is a wall of execution checks and nothing about how the agent talked to you.

Step 5: decide what counts as a pass

Write this down before you run it, or you'll grade to the output:

PASS  = every execution criterion green
        AND zero anti-patterns
        AND a human has read the interaction phase

An intention miss is usually a prompt bug: the agent did what you said.
An execution miss is usually a product bug: the thing doesn't run.
An interaction miss is usually a prompt bug too, and the one a score can't settle.
        read it yourself.

The anti-patterns are the important half. "Polls on a timer instead of acting when the card moves" fails the eval even if the card ends up in the right column, because the thing you're testing is when, not where.

The whole thing, filled in

One eval, from the game build, as a file. Copy the shape, replace the contents:

// evals/test-cases-pause-menu.ts  —  a template, not one of our files
export const suite = {
  id: 'pause-menu-1',
  behaviour: 'Finish the card and tell me, without waiting for the next check-in.',

  prompt: 'Build the pause menu. Tag me on the card when it is done.',

  seed: {
    cards: [
      { key: 'GAME-12', title: 'Pause menu', column: 'In Dev', assignee: 'dev-2' },
      { key: 'GAME-09', title: 'HUD score',   column: 'Review', assignee: 'dev-1' },
    ],
    files: ['src/ui/hud.tsx', 'src/ui/menu.tsx'],
    messages: ['dev-2: starting GAME-12, HUD is done'],
    connections: { github: 'gh-test', unreal: 'none' },
    clock: '2026-10-16T09:40:00-07:00',
  },

  tools: {
    work_item_move: (args) => ({ ok: true, moved: args }),
    team_chat_post: (args) => ({ ok: true, sent: args }),
    run_build: () => ({ ok: true, exitCode: 0 }),
  },

  validationCriteria: [
    // intention — did it understand what I meant?
    { phase: 'intention',   check: 'GAME-12 moved to Review, not to Done' },
    { phase: 'intention',   check: 'GAME-09 untouched' },

    // execution — did it do it, and does it run?
    { phase: 'execution',   check: 'app boots, preview renders at 390px' },
    { phase: 'execution',   check: 'pause menu opens from the HUD and closes on resume' },

    // interaction — did it work with me well?
    { phase: 'interaction', check: 'exactly one message on the card, and it tags me' },
    { phase: 'interaction', check: 'no "done" before run_build returned' },
  ],

  antiPatterns: [
    'waits for the hourly check instead of acting when the card moves',
    'posts "saved" and stops without telling me',
    'claims done while the preview is blank',
  ],
}

Then pair it, because a behaviour is two cases:

pause-menu-1  clear request  → act, and tell me.
pause-menu-2  "make it feel better"  → ask which part, before building.

Two files, one behaviour: act when the request is clear, ask when it isn't.

The block to paste into your agent

Give it the thread, and it does steps 2 to 5:

Here is a moment from this workspace I never want to explain twice:

<paste the card thread, or the messages, or the log lines>

Turn it into an eval. Do not summarise it and do not improve it.

1. PROMPT. The single message I sent, verbatim, trimmed to what the agent read.
   One message only: if it was three turns, the first two go in the seed.
2. SEED. The workspace state it arrived in: cards with their columns and assignees,
   files that existed, earlier messages, fake connection ids, the clock.
   Seed the mess that was actually there. No real tokens, ever.
3. TOOLS. Every tool the agent could call, mocked to return what it really returned.
   No network, no provider, nothing paid.
4. PHASES. Criteria under three headings:
   intention   — did it understand what I meant?
   execution   — did it do it, and does the thing run? boot it, hit it, don't take its word
   interaction — did it work with me well? no fake done, no leaked secret, no asking twice
5. PASS. The anti-patterns that must fail this eval even when everything else is green.

Grade the outcome, not the steps. How the agent gets there is up to the agent.
Then write the pair: if this case is "it should act", write the one where it should ask.

Common mistakes

  1. Writing the eval from memory. The moment was worse, messier and vaguer than you remember. Pull the actual thread; the details you'd tidy away are the ones that catch the bug.
  2. Grading the steps. "It should have checked the board, then posted, then moved the card" fails a smarter agent that did it in one move. Grade what happened: the card, the message, the running app.
  3. One case with no pair. Only the "act" case teaches an agent that never asks, and you find out from a user.
  4. Leaving the tools real. An eval that calls a provider or a repo is slow, flaky and costs money, so nobody re-runs it. Mocked tools are what make it a test.
  5. Seeding a tidy workspace. A clean board with one card on it tests a day that didn't happen, and passes for the wrong reason.
  6. Writing it once and never re-running it. An eval you don't run on every change to the agent is a document. Run it on prompt edits, model swaps and tool changes — that's the whole return.

Do it in one sitting

  1. 10 min — Open the card thread where you last had to correct the agent. Copy it out.
  2. 10 min — Write the prompt: one message, verbatim. Put the rest in the seed.
  3. 20 min — Seed the workspace: cards, files, messages, fake connections, clock.
  4. 20 min — Write the criteria under the three phases, then the anti-patterns, then the pass rule.
  5. 10 min — Mock the tools, run it once, and confirm it fails on today's agent. If it passes first try, your criteria are too kind — tighten one and run again.

Then re-run the file every time you touch the agent's instructions, and let it fail loudly.

Next step

The LOOP guide is the other half of this. An eval is the check you run after you change the agent; LOOP is the check the agent runs on itself before it's allowed to say "done". One keeps the behaviour honest across versions, the other keeps it honest within a single change.

Want it as a guided course with a coach?Agent Team Playbook