Matthew Huntsberry

Field guide

Field guide · 05

Did the agent follow the system? Measure it.

An AI agent can write code that compiles, passes lint and looks finished, and still ignore the design system. We ran a small test, with and without component contracts, and planted mistakes to find out which checks tell the difference.

00

Valid is not the same as adherent

Types and lint tell you the code is valid. They don't tell you the agent used the right component, the right props and the right tokens, that the result matches the design, or that someone can use it with a keyboard. That second set of questions is adherence.

A component contract is one file per component that says what correct use looks like. A checker reads the contracts and validates code against them, with no AI involved. We wanted to know two things: does giving these to an agent change what it builds, and do they catch mistakes the usual tools miss?

01

The test

We used an enterprise design system and gave agents two tasks: a form screen, scored against its Figma frame, and a component-heavy panel, scored against a written spec.

Each task ran in three setups: the system as it stood, the system with improved tokens and styles, and that same version plus contracts, the checker and a paragraph telling the agent to use them. We ran each setup on models from cheap to highly capable. Some runs also had a documentation server, and the results below combine them. A few runs that were given the server and never used it were left out.

Every agent got the same brief and could read all the code in its copy of the repo. We removed our own finished screens so there was no answer to copy.

What we measured

MeasureHowIndependent of the contracts?
Match to the designRender at a fixed size and compare with the Figma frame, pixel by pixel, against a fixed threshold.Yes
Browser checksTest the result in a real browser, including accessibility links between elements, and keyboard use.Yes
Contract errorsRun the checker on the code each agent left behind, in every setup.No. Agents with contracts run this check themselves.
Planted mistakesBreak a clean screen one way at a time and see which checks notice. No agent involved.We wrote the mistakes, so they lean towards what contracts catch.

02

What agents built

On both independent measures, the runs with contracts passed far more often. "Without them" combines the first two setups.

Runs that passed, on the two independent measures

Screen matches the design

  • With contracts and the checkerEvery run
  • Without themAbout half

Panel passes every browser check

  • With contracts and the checkerNearly every run
  • Without themAbout 1 in 5
Rounded, from a small test. Neither score depends on the contracts: one compares pixels with the design file, the other tests the panel in a real browser.

The checker's own score went the same way: every run with contracts finished with no contract errors, against fewer than 1 in 5 without. That is the weakest evidence here, because those agents ran the same check that scored them. It's why the two independent measures matter.

Improved tokens and styles on their own showed no clear improvement across the two tasks.

03

The cheapest model closed the gap

On the panel, the cheapest model went from no passing runs to all of them. The most capable model, without contracts, passed about half.

Panel runs that passed every browser check

  • Cheapest model, without contractsNone
  • Most capable model, without contractsAbout half
  • Cheapest model, with contractsEvery run
Rounded, from only a few runs per model.

Contracts made each run cost more, because the agent reads them and runs the checker. Even so, the cheapest model with contracts cost far less per run than the most capable model without. With this few runs, we wouldn't pick a model from these results. They do suggest that a contract can matter as much as the choice of model.

04

Planting mistakes

Agent runs vary, so we also tested the checks directly. We took one clean screen, planted mistakes one at a time, and asked each check whether it noticed. There is no agent in this test, and it gives the same answer every run.

Planted mistakes each check noticed

  • Types or lintAbout 1 in 6
  • Render compared with the designAbout 2 in 5
  • Contract checkerAbout 3 in 5
  • Checker or renderAbout 5 in 6
Rounded. Mistakes were planted one at a time in a clean screen.

TypeScript and ESLint aren't failing here. They weren't built to know that a prop is discouraged, that a colour should be a token, or that a button should be filled. The biggest gaps were deprecated props and raw values where a token belongs: types and lint missed those, and the contract checks caught them.

Some mistakes got past everything, including a table that lost its accessible name and a select-all that didn't select. That last one is a behaviour bug, and behaviour needs unit tests. No contract check replaces them.

05

Checking the checker

A checker is only useful if its answers can be trusted, so it gets tested too. Each rule has a stable code and a test case built to make that code fire. To test whether it was repeatable, we ran it several times over all of the agents' code. No output changed.

It also had a bug worth knowing about. It once skipped imports from one path, so some runs scored 0 errors on 0 component usages. They actually had errors, and we corrected them. A run with 0 usages hasn't passed. It hasn't been checked.

06

What this doesn't show

This was a small test, with few tasks and only a few runs per combination. The proportions above are rounded and would move with more runs, so read them as a direction, not as rates. We also built the test ourselves: the contracts, the checker, the tasks and the mistakes. Nobody outside the team has repeated it.

The setup with contracts is a bundle. It includes the checker and a paragraph telling the agent to use them, so these results can't credit the contract file alone. The pixel score is blunt too: a single tint across a table can swing a screen's score. And agent behaviour on real product work, over time and with real reviewers, wasn't part of this.

07

The test

Take away the agent's opinion of its own work. What number is left?

If you want to measure this on your own system, start with one score the contracts can't influence, such as a render against the design or a browser check. Plant mistakes to see what your current tools miss. Then run your checks more than once on the same code, because a check that changes its answer can't be used to measure anything.