Guardrails, approvals and evals

You change one line in your agent’s system prompt to make it friendlier. You try it on the example you always use, and it’s great. You ship it. Next week, finance asks why a sofa was refunded without anyone signing it off.

Nothing in that story was careless. The demo just couldn’t see the problem. This lesson is about the two things that could: guardrails, which limit what the agent can do whatever it decides, and evals, which tell you whether a change made it better or worse.

Guardrails live in code

A guardrail is a check that runs outside the model, in the loop you wrote, every time. The common ones:

• Limits on steps, time and money, so a confused agent can’t run forever. Lesson one’s step limit was one.
• Approval gates: a person confirms before anything risky runs. OWASP’s advice for agents is to “require a human to approve high-impact actions before they are taken”.
• Hard rules in the tools themselves: a refund function that refuses anything over EUR 100 without a staff ID, whatever argument it receives.
• Logs of every model call, tool call and result - the traces the widgets in this track show. When something goes wrong, the log is how you find out why.

A sentence in the prompt saying “never refund more than EUR 100” is not a guardrail. It’s a request. The model may honour it - until a prompt change, a new model version or an unusual email nudges it the other way.

Evals: tests for behaviour

An eval is a fixed set of test cases, each with a check that says pass or fail, run every time you change the agent. Anthropic’s guidance on building them:

• “Be task-specific: Design evals that mirror your real-world task distribution. Don’t forget to factor in edge cases!”
• “Automate when possible: Structure questions to allow for automated grading” - multiple choice, string match, checks in code.
• “Prioritize volume over quality: More questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals.”

For an agent, the most useful checks are about what it did, not how its reply sounds. Did the refund tool run? Was the reply in Spanish? Did any other customer’s address appear? Code can grade all of those, so the whole suite runs in seconds.

1. 1Anything that can change behaviour counts as a change: a prompt edit, a new model version, a new tool, a new description.
2. 2The whole eval set runs, not just the example you had in mind. Each case is graded by code, so it takes seconds.
3. 3One failing case blocks the change. That is the point: the eval sees the cases you were not thinking about.

The demo was fine

The simulated agent below answers support emails and can refund orders. Prompt v1 is cautious and sends all refunds to staff. Prompt v2 was written to fix two complaints: customers who write in Spanish get English replies, and small refunds wait days for staff. The eval set has eight cases; the demo is the first one.

Look at what happened:

• The demo passed for every version. It asks where an order is, which every version handles. A demo can only show you the case you chose.
• v1 failed two cases: the Spanish speaker and the EUR 40 refund. Those are the complaints v2 was written to fix.
• v2 fixed both, and broke two others. “Resolve problems without bothering staff” also meant refunding a EUR 650 sofa, and obeying the injected bulk refund from the tools lesson.
• Going back to v1 doesn’t help. It brings the first two failures back. The fix that keeps v2’s improvements is a guardrail in code: refunds over EUR 100 go to staff, whatever the prompt says. With it, all eight pass.