Build and debug AI agents
Generated, then checked by the eval suite. Last read by a person on 2026-10-12, 1 day ago. Its facts were checked by the eval suite on 2026-10-12.
You are learning to build or debug an AI agent and want practice whose help fades as you improve, or you guide learners and want tasks with a help level you set, worked versions to demonstrate from, and the evidence behind each design choice.
Practice tasks that teach engineers to build an AI agent on the Claude API, to work out what it costs to run, and to find why a broken one fails. Help shrinks as skill grows, while the tests stay the same at every level. One learner has used them so far, and results across learners have not been measured.
Open the practice tasks. The rest of this page says what the design is meant to change, how strong its evidence is, and what has not been shown yet.
At a glance
- What a learner can do afterward. Build a working agent, say what each design choice costs in requests to the model and in tokens, find the cause of a broken agent from its evidence, and explain those answers out loud.
- Who it is for. Learners building AI agents, and the facilitators who guide them. Anyone can set the help level, so a facilitator can show a worked version to a group, then use the prediction tables and the broken agents as discussion prompts.
- How strong the evidence is. The learning choices rest on published studies, most with school and university students and one in an industrial training course. That the findings carry over to engineers is our assumption. The screen-design choices rest on design guidance and essays, not studies.
- What is not shown yet. One learner, who also commissioned the tasks, has used them, so there are no results across learners yet. An AI model checked that each source says what its sentence claims, and the owner then reviewed and approved every verdict.
- What it costs. The practice pages are free and run in a browser. The graded feedback on a learner's code and explanation needs access to the project's repository and a Claude API key, and the API bills per use. The cost per feedback run and the time a task takes have not been measured yet.
Objections
What a skeptical reader might say, and the answer.
"People learn by being thrown in; worked examples are spoon-feeding." Worked examples are the first step, not the whole course. Help that aids beginners can get in the way of people who know more, so it is taken away step by step until the learner builds and explains it alone. In algebra studies, students who studied solved examples later solved problems of the same type faster and with fewer errors than students who only practiced solving, though the gain held only for problems built the same way. In later studies, removing solved steps one at a time helped learners solve similar problems better than alternating examples with problems. Our choice: from level 2, the broken-agent task asks what evidence you would request before it shows any choices, much as an on-call engineer meets a failure.1
"An AI tutor or chatbot makes structured practice unnecessary." Students who practiced recalling a passage remembered more of it two days and a week later than students who reread it, though rereading did better on a test five minutes later. Our view, which the study did not test: reading a chatbot's answer is closer to rereading than to recalling. Here the learner writes the agent and the explanation first, and a model then grades only what they wrote against five fixed criteria, quoting the line each finding is about.2
The rest of this page is the method: the need, the gaps and the principles behind the design.
Needs analysis
Goal. Build a Claude agent, say what each design choice costs in requests and tokens, find why an agent misbehaves from its evidence, and defend those answers out loud in an interview.
Who it is for. Built for, and so far used by, one engineer preparing for AI engineer interviews (Phil). The pages are written so that a non-technical colleague, a beginner programmer and a working engineer can each follow them. Before a page ships, an AI model reads it as each of those readers and flags anything they could not follow.
How the need was established:
- The interview for the role was mapped to six skill areas (software engineering, programming language, API, AI, mathematics and theory, production and scale). Each task is tagged to the areas it trains, so an area no task covers shows up as a gap. (Record: the learning map on the practice home page)
- Interviews for the role ask candidates to find why an agent misbehaves, for example one stuck calling the same tool. Task 2 was built after that came up in a real interview: a task is added when a need shows up, not in advance. (Record: progress/journal.jsonl, entry 387f45)
- On his first try, the learner's code failed the task 1 tests, which pointed to the gap task 1 was built for. (Record: progress/journal.jsonl, entry 28f065)
Gaps
What the learner could not yet do, the evidence for it, and what the design does about it.
Saying what an agent costs to run
Evidence. Each request to the model is billed and adds wait time, and fewer tool calls does not always mean fewer requests. On his first try, the learner's code failed the task 1 tests, which count the requests, round trips and tokens of each version of the same agent.
Response. Task 1 shows four working versions of the same agent, built four ways, and tests fix each version's counts. The learner studies them before writing their own: in algebra studies, students who studied solved examples first later solved similar problems faster and with fewer errors. Our choice: the learner also predicts each version's counts before the measured answers appear.3
Finding why an agent misbehaves from its evidence, not the story about it
Evidence. A real interview asked for exactly this.
Response. Task 2 draws practice cases from a bank of over 2,000 generated broken agents. Each case makes the learner recall a diagnosis rather than reread one; in studies, practicing recall kept material in memory longer than rereading did. The score explains each criterion rather than only marking it right or wrong, one of the kinds of feedback a review of the research describes. Our choices: from level 2 it asks what evidence you would request before it shows any choices, and it replays the fixed agent beside the broken one.4
Principles
The principles behind the design and the evidence for each. Its note under Sources says what kind of evidence that is: a study, a review, design guidance or product documentation.
Learning design
- Worked examples first. Each task opens with worked versions of one program, measured by tests, before the learner writes their own. Searching for a solution uses up attention a learner could otherwise spend learning how such problems are built, so studying solved problems comes first. It is the first step, not the whole course.5
- Help that fades, a bar that does not. Four help levels (0 watch, 1 fill, 2 build, 3 explain) hide worked code and hints as the learner advances, and the tests and feedback are the same at every level. Help that aids beginners can get in the way of people who know more, so it is removed step by step. Our choice: the learner sets the level, so an experienced engineer can start higher; these studies did not test that.6
- Predict before the reveal. Every version's cost is predicted in a table before the measured answers appear. In the cited study, predicting an outcome before seeing it led to more learning than judging it afterward, and stronger surprise at a wrong prediction went with more learning. Our choice: the worked diagram and the counting rules stay above the table, so a prediction can be reasoned rather than guessed.7
- Explain it in your own words. At level 3 the learner writes out why each version costs what it does. Students who did well explained worked examples to themselves far more than students who did poorly, and prompts to explain each step helped on both similar and new problems. Our choice: a model grades the written explanation against fixed criteria, because an interview asks for one; neither study graded a written explanation.8
- Practice with feedback that says why. Practice runs and diagnosis cases are recorded in a journal, and feedback marks each criterion as met or not yet and gives the reason. Research on how experts get good stresses practice with quick, informative feedback, recall practice aids memory, and a review of feedback research advises that feedback be specific. Our choice: each finding cites the source it rests on.9
User experience
- A model before a paragraph. The round-trips lesson is a model the reader can operate, and each figure starts from one concrete run before the general rule. The designer Bret Victor argues in two essays that readers understand an idea more deeply when they can operate it and move between a concrete run and the general rule; these are design arguments, not studies. Our choice: prose never introduces a mechanism.10
- Solve in one view. Everything a step needs is on screen at once: long content scrolls inside its own pane, and a question never covers the evidence it asks about. Learners do better when parts that explain each other sit together, and design guidance adds that people should not have to remember something from one screen to use it on another. An automatic check plays every step of the diagnosis page and fails any at which the page itself scrolls.11
- Disclosed in layers. A figure's first view has no numbers, and the counts open in place when the reader asks. Design guidance advises showing the main items first and the rest on request, and warns that more than two layers usually hurts usability. Our choice: the exact messages open as a third layer anyway, closed until opened, for readers who want the arithmetic.12
- Plain words, from the reader's seat. Pages use the reader's everyday words rather than insider terms, as design guidance advises. Our choice: a non-technical colleague is one of the readers pages are checked as, and must be able to find themselves in the who-does-what figure.13
- Built for phone and desktop. Each screen size gets its own layout rather than a phone layout stretched onto a desktop, as design guidance advises, so a learner can practice on either. Automated checks render every page at phone and desktop width and catch layout faults.14
Engineering
- The real library against a stand-in model. Practice runs and tests use Anthropic's official Python library against a simple stand-in that follows Anthropic's published tool-use rules, so results repeat exactly and cost nothing. The pages say it is a stand-in, not a model. Our choice: only the network layer is replaced, and a run against the real model is required before any version counts as done.15
Sources
Each numbered note names the source, what kind of evidence it is, who it studied when the source says, what in it supports the sentence, and who checked it. "Supported" means the checker found that the source says what the sentence claims. It does not mean the finding has been shown to hold for this audience.
Kalyuga, S., Ayres, P., Chandler, P., & Sweller, J. (2003). The expertise reversal effect. Educational Psychologist, 38(1), 23-31.; Sweller, J., & Cooper, G. A. (1985). The use of worked examples as a substitute for problem solving in learning algebra. Cognition and Instruction, 2(1), 59-89.; Renkl, A., Atkinson, R. K., Maier, U. H., & Staley, R. (2002). From example study to problem solving: Smooth transitions help learning. The Journal of Experimental Education, 70(4), 293-315.; Atkinson, R. K., Renkl, A., & Merrill, M. M. (2003). Transitioning from studying examples to solving problems: Effects of self-explanation prompts and fading worked-out steps. Journal of Educational Psychology, 95(4), 774-783.
Supported. Experiments and a review. In algebra experiments, studying solved examples instead of solving problems led to faster solving with fewer errors, but only on problems built the same way (Sweller and Cooper). Later experiments found removing solved steps one at a time beat example-problem pairs on similar problems (Renkl; Atkinson). A review finds beginner help can hurt skilled learners (Kalyuga). More information. Checked by Phil Wagner, 2026-10-09. Back to the text
-
Supported. Two experiments with students who read prose passages. Students who practiced writing down what they recalled remembered more two days and one week later than students who reread. At five minutes, rereading did better. The study did not test chatbots. More information. Checked by Phil Wagner, 2026-10-09. Back to the text
-
Supported. A series of experiments with algebra students. Students who studied solved examples took less time than students who solved the same problems, then solved later problems of the same type faster and with fewer errors. The gain held only for problems built the same way, which matches "similar problems". More information. Checked by Phil Wagner, 2026-10-09. Back to the text
Roediger, H. L., & Karpicke, J. D. (2006). Test-enhanced learning: Taking memory tests improves long-term retention. Psychological Science, 17(3), 249-255.; Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153-189.
Supported. Carried from verdict 11e5bc159ed2 (Phil Wagner, 2026-10-09): the cited sentences are word for word the same, and only the project's own bank count changed, from 2,049 to over 2,000 (bank.jsonl holds 2,231). Two sources. An experiment with students: practicing recall beat rereading on tests two days and one week later (Roediger and Karpicke). A review of feedback research lists kinds of feedback, including telling a learner whether they were right and explaining the correct answer (Shute). The bank size and the level 2 design are the project's own. More information. Checked by Phil Wagner (verdict 11e5bc159ed2, carried), 2026-10-12. Back to the text
Sweller, J., & Cooper, G. A. (1985). The use of worked examples as a substitute for problem solving in learning algebra. Cognition and Instruction, 2(1), 59-89.; Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257-285.
Supported. A theory paper with a computer model and an experiment (Sweller 1988): working toward a goal by trial and search uses up mental effort that could go to learning how problems are built. Algebra experiments (Sweller and Cooper) found studying solved examples led to faster, more accurate solving of problems of the same type. More information. Checked by Phil Wagner, 2026-10-09. Back to the text
Renkl, A., Atkinson, R. K., Maier, U. H., & Staley, R. (2002). From example study to problem solving: Smooth transitions help learning. The Journal of Experimental Education, 70(4), 293-315.; Atkinson, R. K., Renkl, A., & Merrill, M. M. (2003). Transitioning from studying examples to solving problems: Effects of self-explanation prompts and fading worked-out steps. Journal of Educational Psychology, 95(4), 774-783.; Kalyuga, S., Ayres, P., Chandler, P., & Sweller, J. (2003). The expertise reversal effect. Educational Psychologist, 38(1), 23-31.
Supported. A review (Kalyuga and colleagues) finds that help which works for beginners can lose value or hurt more skilled learners. Experiments by Renkl and by Atkinson found that removing solved steps one at a time helped. None of them tested letting the learner pick the level. More information. Checked by Phil Wagner, 2026-10-09. Back to the text
-
Supported. Two tasks with eye tracking. Predicting an answer before seeing it, rather than judging it afterward, helped participants learn geography facts. Surprise, measured by pupil size, showed up only after a prediction, and stronger surprise went with more learning. More information. Checked by Phil Wagner, 2026-10-09. Back to the text
Chi, M. T. H., Bassok, M., Lewis, M. W., Reimann, P., & Glaser, R. (1989). Self-explanations: How students study and use examples in learning to solve problems. Cognitive Science, 13(2), 145-182.; Atkinson, R. K., Renkl, A., & Merrill, M. M. (2003). Transitioning from studying examples to solving problems: Effects of self-explanation prompts and fading worked-out steps. Journal of Educational Psychology, 95(4), 774-783.
Supported. Two sources. Chi and colleagues recorded students thinking aloud over solved physics problems: strong students explained the steps to themselves far more than weak ones, a link, not proof of cause. In Atkinson's experiments, prompts to name the rule behind each step, given while solved steps were removed one at a time, helped on similar and differently built problems. More information. Checked by Phil Wagner, 2026-10-09. Back to the text
Ericsson, K. A., Krampe, R. T., & Tesch-Romer, C. (1993). The role of deliberate practice in the acquisition of expert performance. Psychological Review, 100(3), 363-406.; Roediger, H. L., & Karpicke, J. D. (2006). Test-enhanced learning: Taking memory tests improves long-term retention. Psychological Science, 17(3), 249-255.; Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153-189.
Supported. Three sources. Ericsson's paper on expert performance, built on studies of musicians, says focused practice needs quick, informative feedback. An experiment with students found recall practice improved memory days later. A review of feedback research advises feedback that is specific. More information. Checked by Phil Wagner, 2026-10-09. Back to the text
Victor, B. (2011). Explorable explanations.; Victor, B. (2011). Up and down the ladder of abstraction.
Supported. Two design essays by Bret Victor, not studies, as the sentence says. One argues that documents readers can operate make them active readers and build understanding. The other moves between one concrete simulated run and the general view of a system. More information. Checked by Phil Wagner, 2026-10-09. Back to the text
Chandler, P., & Sweller, J. (1991). Cognitive load theory and the format of instruction. Cognition and Instruction, 8(4), 293-332.; Nielsen, J. 10 usability heuristics for user interface design. Nielsen Norman Group.
Supported. Experiments by Chandler and Sweller, with training in electrical work and biology: learners did better when parts that only make sense together were merged into one place. A Nielsen Norman Group design guideline says users should not have to remember information from one part of a screen to use it in another. The scroll check is the project's own. More information. Checked by Phil Wagner, 2026-10-09. Back to the text
Nielsen, J. (2006). Progressive disclosure. Nielsen Norman Group.
Supported. A design guideline article from Nielsen Norman Group. It says to show users only a few of the most important options first and offer more on request, and that designs with more than two layers usually have low usability. This page uses a third layer anyway. More information. Checked by Phil Wagner, 2026-10-09. Back to the text
Nielsen, J. 10 usability heuristics for user interface design. Nielsen Norman Group.
Supported. A design guideline from Nielsen Norman Group. Its second rule of thumb says a design should speak the users' language, with words and ideas familiar to them rather than insider jargon. The non-technical reader check is the project's own choice and is labeled that way. More information. Checked by Phil Wagner, 2026-10-09. Back to the text
Budiu, R., & Pernice, K. (2016). Mobile first is not mobile only. Nielsen Norman Group.
Supported. A design guideline article from Nielsen Norman Group. It says to design for every device, because devices differ in screen size and how people use them, and warns that designing once and reusing that design can give a poor experience. The automated checks are the project's own. More information. Checked by Phil Wagner, 2026-10-09. Back to the text
Anthropic Python SDK; Handle tool calls
Supported. Product documentation. Anthropic publishes the Python library in its own GitHub account. Its "Handle tool calls" page sets the rules a stand-in can follow, such as each tool result coming right after its request and first in the reply. The stand-in and the live-run rule are the project's own choices. More information. Checked by Phil Wagner, 2026-10-09. Back to the text