Claude Code Can Now Test and Improve the AI Tools You Build
If you’ve built a tool that calls Claude, you’ve probably asked yourself whether it’s any good. Most of us answer by trying a few inputs and deciding it looks fine. Anthropic has added two commands to Claude Code that replace that guess with a score.
What changed
On 28 September 2026 Anthropic added /claude-api build-eval and /claude-api hillclimb to the claude-api skill for Claude Code. You install the skill from Anthropic’s skills repository, then run the commands inside Claude Code.
Two terms first. An eval is a set of test cases with a way to mark each answer right or wrong. Hillclimbing means changing one thing at a time and keeping the change only if the score goes up.
build-eval walks you through building the eval. It gathers test inputs from real usage, bug reports and cases you write yourself. It helps you choose how answers get graded, either by code or by a model acting as the grader. Then it checks that the eval itself is sound, because a broken test gives you a confident wrong answer.
hillclimb improves your tool against that eval. It proposes one change at a time, to your prompts, skills, tools or model settings. It also splits the test cases, so the tool can’t pass by memorising the answers.
The numbers in Anthropic’s post are worth reading closely. On a customer-support test set, accuracy went from 74.4% at 4.6 cents per ticket to 88.9% at 1 cent per ticket, using Sonnet 5 with better prompting. The tool got more accurate and cheaper at the same time. Anthropic also ran the process on the claude-api skill itself, and its pass rate rose from 66% to about 88% over 24 rounds.
Who it helps
People who have already built something. If you followed the Weekend Build Pack and now have a tool you use every week, this is the next step: finding out how often it’s wrong, and fixing that without guessing.
It suits tools that do one job repeatedly with answers you can check. TARA, the road appraisal tool in my case study, is that kind of tool. [KYE: have you tried build-eval on TARA? one line on what happened]
The post is honest about the limits. A tool can be tuned to pass the test without getting better in real use. That’s called overfitting. And if your scores swing a lot from one run to the next, the swing can hide problems with the measurement itself. A score is only as good as the test cases behind it.
One thing to try today
Pick one task your tool does. Write down 10 real examples and what a good answer looks like for each. Use real inputs from the last month, including at least one that went wrong.
That list is the start of an eval, with or without the new commands. Run your tool on all 10 and count how many it gets right. Now you have a number to beat.
Sources
How this was written: Drafted with AI (Claude), edited and verified by the author.