Curriculum auditor

This tool reads a lesson and finds which XQ skills it gives students a chance to practice. It also scores student work on those skills. Claude, an AI model, makes each decision. Then regular code checks it before it counts: every quote must really be in the lesson, every skill must be real, and every math answer is worked out again by code.

Who it's for. The tool is for teachers and curriculum teams who want to check a lesson against the XQ Competencies and get a first pass on student work. The notebook is for people learning to build AI agents: it teaches how the tool was built, with exercises. Students never use it. Their work is what gets scored.

This page is the project's notebook. It shows every step and every result, using saved recordings of real API calls.

What it found

• Errors in the answer key. The test lesson had three planted mistakes. Code found two: a calculator set to radians (6.15 m instead of 13.22 m) and a missing 1.5 m. Claude found the third: cosine used where sine was needed. Claude also found a mistake nobody planted.
• Claude's scores compared with mine. I scored 16 student responses on 2 skills each before seeing Claude's scores. When we both gave a level, we were never more than one level apart. Exact matches ranged from 86% to 91% across three runs. The main disagreement: on 5 scores where the student did only a little work, such as a bare "0.31" or a one-line answer, Claude said the work showed too little to score, and I gave it Level 1. Where there was no work at all, we agreed on 0.
• Claude does not always agree with itself. I had Claude score the same work three times. 25 of 30 scores came out the same every time. Across six runs in all, one agreement measure ranged from 0.75 to 0.96, so one run's numbers are not stable.

Limitations

This shows the method. It does not prove the tool works on real student work.

1. One rater. Only one teacher scored the work, so there is no measure of how often two teachers agree. Without that baseline, nobody can say whether Claude's agreement is good or bad.
2. The student work is synthetic, and Claude wrote it. Real students write differently, and Claude may find its own writing easier to score.
3. The sample is small. 16 responses and 32 scores. Changing one or two scores moves the results a lot.
4. Coverage was never checked against a person. Only the scores were compared with a teacher. Nobody checked Claude's calls on which skills a lesson teaches.
5. One lesson, one subject, four skills. The results may not hold for other subjects, grade levels, or XQ skills.

The full list of ten is in the README.

Where to look

• Section 2: the agent loop by hand, where the math check catches the radian-mode error
• Section 4: Claude judges, code verifies, where the validators reject a fabricated quote, a missing skill, and copied instructions
• Section 4b: Your turn, four exercises where you write the checks yourself
• Section 6: the agreement report, with every pair and all three runs
• Source, tests, and setup on GitHub

Executed in replay mode on 2026-09-30, from recordings of real Claude Opus 5.5 calls.

---

Build a curriculum auditor: Claude judges, code verifies

This notebook builds a tool that reads a lesson, finds which skills it gives students a chance to practice, and scores student work. You build it one piece at a time.

Claude, an AI model, makes each judgment. Plain Python code checks each one before it is accepted:

• Every quote has to appear, word for word, in the right text.
• Every skill ID has to be real.
• Every math answer is worked out again by code that never asks the model.

By the end, you will be able to:

1. Describe a tool to Claude and handle one tool call.
2. Write an agent loop that sends tool results back until the job is done.
3. Check every model judgment in code before accepting it, and send back fixes Claude can act on.
4. Compare a model's scores with a teacher's, and explain why one run's numbers are not the final word.

You do not need an API key. The notebook replays saved recordings of real calls to Claude Opus 5.5, so it runs offline and costs nothing. Running every cell takes a few minutes. Reading it carefully and doing the exercises takes about two hours.

Section	What you build	Minutes
0	Setup, and why the skill framework is downloaded instead of included	5
1	One tool call	10
2	The agent loop, written by hand	15
3	Looking up skills versus putting them all in the prompt	10
4	Claude judges, code verifies	15
4b	Your turn: write four of the checks yourself	20
5	The same agent with the SDK's Tool Runner	10
6	Scoring student work, and comparing it with a teacher	15
7	The same tools in Claude Code, over MCP	5
8	Optional: a live run on your own lesson	5+

0. Setup

The tool works with two skill frameworks. Both use the same data format, so every line of code below works with either one.

• The demo rubric has 8 skills written for this project. It comes with the code, and every recording in this notebook uses it.
• XQ Competencies has 115 skills, published by XQ Institute in the XQ Competency Navigator. The framework belongs to XQ, so this project does not include its text. Instead, curriculum-auditor fetch downloads a copy to your data/ folder. It checks that nothing is missing (5 outcomes, 37 competencies, 115 skills, 460 level descriptions) and saves a fingerprint, so a later download can tell you whether XQ changed any wording. A check before each commit stops XQ's text from ever being saved to the repository.

import json
import os
from pathlib import Path

ROOT = Path.cwd().parent if Path.cwd().name == "notebooks" else Path.cwd()
os.chdir(ROOT)

from curriculum_auditor.rubric import load_rubric
from curriculum_auditor.segment import load_curriculum
from curriculum_auditor.work import load_responses

demo = load_rubric("demo")
lesson = load_curriculum("content/lesson-heights.md")
responses = load_responses("content/responses.json")

print(demo.title, "|", len(demo.skills), "skills")
for skill in demo.skills[:3]:
    print(" ", skill.id, skill.name)
print()
print(f"Lesson: {len(lesson.sections)} sections")
for section in lesson.sections:
    print(" ", section.id)

xq_ready = (ROOT / "data/xq_framework.json").exists()
print()
print("XQ cache present:", xq_ready, "(run `uv run curriculum-auditor fetch` to add it)")

Demo rubric (original, not XQ) | 8 skills
  DEMO.1.a Checks reasonableness
  DEMO.1.b States angle units
  DEMO.2.a Draws a labeled diagram

Lesson: 7 sections
  measuring-heights-you-cannot-reach
  measuring-heights-you-cannot-reach/overview
  measuring-heights-you-cannot-reach/warm-up
  measuring-heights-you-cannot-reach/task-a-the-ramp
  measuring-heights-you-cannot-reach/task-b-something-tall-outside
  measuring-heights-you-cannot-reach/exit-ticket
  measuring-heights-you-cannot-reach/answer-key

XQ cache present: True (run `uv run curriculum-auditor fetch` to add it)

The lesson in content/lesson-heights.md was written for testing. Its answer key has three planted mistakes:

1. A height worked out with the calculator set to radians instead of degrees.
2. A sum that leaves out one number.
3. The wrong trig ratio for the situation.

Watch for them. Code catches the first two. Only a careful reader can catch the third.

1. One tool call

A tool is a function Claude can ask your code to run. You describe each tool with three things: a name, a short description, and a schema that lists the inputs it takes. When Claude wants a tool, its reply includes a tool_use block with the tool's name and the inputs. Your code runs the function and sends the result back.

Every request in this project uses the same settings, set in one place, request_settings():

• Model: claude-opus-5-5, with effort set to high. Opus 5.5 would use medium otherwise, so the setting is written out.
• Thinking: adaptive, so Claude decides how much to think before it answers.
• Strict tools: strict: true on every tool, so Claude's inputs always match the schema.
• Tool choice: left on auto. Opus 5.5 does not allow forcing a tool call, so the instructions tell Claude which tool to use.
• Caching and fallbacks: prompt caching is on, and fallbacks: "default" sends a refused request to another model automatically.

from curriculum_auditor import prompts
from curriculum_auditor.client import make_client, request_settings, strict_tools
from curriculum_auditor.tools import TOOL_DEFINITIONS, Session, dispatch

print(json.dumps(request_settings(), indent=1))
print()
print([tool["name"] for tool in TOOL_DEFINITIONS])

{
 "model": "claude-opus-5-5",
 "max_tokens": 32000,
 "thinking": {
  "type": "adaptive"
 },
 "output_config": {
  "effort": "high"
 },
 "cache_control": {
  "type": "ephemeral"
 },
 "betas": [
  "server-side-fallback-2026-07-01"
 ],
 "fallbacks": "default"
}

['list_competencies', 'get_descriptors', 'check_answer', 'submit_section_coverage', 'record_score']

RECORDING = "recordings/coverage-lesson-heights-demo-manual.json"
client = make_client("replay", recording=RECORDING)   # swap "replay" for "live" to call the API

answer_key = lesson.section("measuring-heights-you-cannot-reach/answer-key")
session = Session(demo, lesson)
system = prompts.coverage_system(demo)
messages = [{"role": "user", "content": prompts.coverage_message(answer_key)}]

with client.beta.messages.stream(**request_settings(), system=system,
                                 tools=strict_tools(TOOL_DEFINITIONS), messages=messages) as stream:
    reply = stream.get_final_message()

print("stop_reason:", reply.stop_reason)
for block in reply.content:
    print("-", block.type, getattr(block, "name", ""), json.dumps(getattr(block, "input", ""))[:160])

stop_reason: tool_use
- thinking  ""
- tool_use check_answer {"item": {"id": "A3", "mode": "numeric", "expected": "3.6*sin(5)", "actual": "0.31", "unit": "m", "actual_unit": "m", "angle_convention": "degrees", "absolute_t
- tool_use check_answer {"item": {"id": "A4", "mode": "numeric", "expected": "3.6*cos(5)", "actual": "3.59", "unit": "m", "actual_unit": "m", "angle_convention": "degrees", "absolute_t
- tool_use check_answer {"item": {"id": "A4-check", "mode": "numeric", "expected": "sqrt(0.31^2+3.59^2)", "actual": "3.60", "unit": "m", "actual_unit": "m", "angle_convention": null, "
- tool_use check_answer {"item": {"id": "B4", "mode": "numeric", "expected": "15*tan(38)+1.5", "actual": "6.15", "unit": "m", "actual_unit": "m", "angle_convention": "degrees", "absolu
- tool_use check_answer {"item": {"id": "Extension", "mode": "numeric", "expected": "25*tan(25)+1.5", "actual": "11.66", "unit": "m", "actual_unit": "m", "angle_convention": "degrees",
- tool_use check_answer {"item": {"id": "Exit", "mode": "numeric", "expected": "40*cos(55)", "actual": "22.94", "unit": "m", "actual_unit": "m", "angle_convention": "degrees", "absolut
- tool_use get_descriptors {"skill_ids": ["DEMO.1.a", "DEMO.2.c", "DEMO.2.b"]}

Claude asked for several tools in one reply. The thinking block holds its reasoning. On Opus 5.5 the text is hidden by default, but the block still has to be sent back unchanged on the next turn.

Now run one of those tool calls yourself. dispatch finds the tool by name and runs it with Claude's inputs.

first_call = next(b for b in reply.content if b.type == "tool_use")
result = dispatch(session, first_call.name, first_call.input)
print(first_call.name)
print(json.dumps(result, indent=1)[:900])

check_answer
{
 "id": "A3",
 "mode": "numeric",
 "status": "correct",
 "reason": "Compared values within max(absolute_tolerance, relative_tolerance * |expected|).",
 "expected_value": "0.313760673892",
 "actual_value": "0.310000000000",
 "absolute_error": "0.00376067",
 "allowed_error": "0.00500000",
 "unit": "m",
 "angle_convention": "degrees",
 "warnings": []
}

2. The loop by hand

An agent is that exchange, repeated:

1. Send the conversation to Claude.
2. Run the tools Claude asks for.
3. Send the results back.
4. Repeat until the job is done.

The loop below is the whole agent. curriculum_auditor/loop.py has the same code with a few extra ways to stop.

Three rules keep it working:

1. Never edit the history. Each reply goes back exactly as Claude sent it, thinking blocks included. Opus 5.5 rejects a conversation whose earlier turns were changed.
2. Answer every tool call, all in one message. A rejected call still gets an answer: the list of fixes, marked is_error.
3. Code decides when the job is done. Here, it is done when the section's coverage is accepted.

from curriculum_auditor.tools import ToolRejected

client = make_client("replay", recording=RECORDING)   # a fresh replay client starts from the first request
session = Session(demo, lesson)
messages = [{"role": "user", "content": prompts.coverage_message(answer_key)}]

for turn in range(1, 10):
    with client.beta.messages.stream(**request_settings(), system=system,
                                     tools=strict_tools(TOOL_DEFINITIONS), messages=messages) as stream:
        reply = stream.get_final_message()
    messages.append({"role": "assistant", "content": reply.content})

    results = []
    for block in reply.content:
        if block.type != "tool_use":
            continue
        try:
            output = dispatch(session, block.name, block.input)
            results.append({"type": "tool_result", "tool_use_id": block.id, "content": json.dumps(output)})
            print(f"turn {turn}: {block.name} accepted")
        except ToolRejected as rejected:
            results.append({"type": "tool_result", "tool_use_id": block.id,
                            "content": rejected.message(), "is_error": True})
            print(f"turn {turn}: {block.name} REJECTED: {rejected.summary}")

    if session.sections[answer_key.id].status != "pending" or not results:
        break
    messages.append({"role": "user", "content": results})

print()
print("Section status:", session.sections[answer_key.id].status)

turn 1: check_answer accepted
turn 1: check_answer accepted
turn 1: check_answer accepted
turn 1: check_answer accepted
turn 1: check_answer accepted
turn 1: check_answer accepted
turn 1: get_descriptors accepted
turn 2: check_answer accepted
turn 3: submit_section_coverage accepted

Section status: accepted

Claude checked the answer key with check_answer. That tool never uses the model. It reads each expression, allows only a short list of math operations, works out the value, and compares it with the key's answer within a stated tolerance. It never runs the text as code, so an answer key cannot run anything harmful.

for check in session.answer_checks:
    r = check["result"]
    print(f"{str(r.get('id')):28} {r['status']:13} expected {r.get('expected_value')}  key says {r.get('actual_value')}")

A3                           correct       expected 0.313760673892  key says 0.310000000000
A4                           correct       expected 3.58630091313  key says 3.59000000000
A4-check                     unsupported   expected None  key says None
B4                           incorrect     expected 13.2192843976  key says 6.15000000000
Extension                    incorrect     expected 13.1576914539  key says 11.6600000000
Exit                         correct       expected 22.9430574540  key says 22.9400000000
A4-check                     correct       expected 3.60335954354  key says 3.60000000000

One check came back unsupported. Claude wrote 0.31^2, but the checker only accepts ** for powers. The checker refused rather than guess, Claude read the reason, and the retry passed.

Now the planted mistakes:

• B4 is the radian mistake. 15 × tan(38°) + 1.5 is 13.22 m. The key says 6.15, which is what you get with the calculator in radians.
• The extension leaves out a number, so its answer does not match its own expression.
• The exit ticket passes, because its arithmetic is right. Its mistake is the setup: it uses cosine where the height needs sine. The checker cannot see setup mistakes.

Claude can. The coverage tool has a notes field for anything a teacher should check.

print(session.sections[answer_key.id].claude_notes)

Answer-key errors (checked in degrees):
1) B4 is incorrect. 15 × tan(38°) + 1.5 = 13.22 m, not 6.15 m. The 6.15 comes from calculating tan(38) in radians mode.
2) The Extension is incorrect. 25 × tan(25°) + 1.5 = 13.16 m, not 11.66 m. The 11.66 is 25 × tan(25°) alone, so the 1.5 m eye height was left off even though the setup includes it.
3) Exit ticket: the arithmetic is right (40 × cos(55°) = 22.94), but check the setup. If 40 is the hypotenuse (for example a string or ladder length) and 55° is the angle of elevation from the ground, the height is the opposite side: 40 × sin(55°) ≈ 32.77 m. Cosine gives the horizontal distance instead. The problem text is not in this section, so this needs a teacher check.
A3, A4, and the A4 Pythagorean check are all correct.

The checker never guesses whether angles are in degrees or radians. If you leave angle_convention out, it refuses. If you give it radians for a degree problem, it does what you asked and adds a warning.

from curriculum_auditor.mathcheck import check_answer

base = {"id": "demo", "mode": "numeric", "expected": "sin(30)", "actual": "0.5",
        "unit": "dimensionless", "actual_unit": "dimensionless", "absolute_tolerance": "0.001"}
for convention in (None, "degrees", "radians"):
    r = check_answer({**base, "angle_convention": convention})
    print(f"{str(convention):8} -> {r['status']:12} {r.get('warnings') or r.get('reason', '')}")

None     -> unsupported  This expression uses trig. Set angle_convention to 'degrees' or 'radians'.
degrees  -> correct      Compared values within max(absolute_tolerance, relative_tolerance * |expected|).
radians  -> incorrect    ['sin(30) is read as 30 radians. Check whether the key meant degrees.']

3. Looking up skills versus putting them all in the prompt

Claude needs each skill's level descriptions to judge it. You can provide them in two ways:

• Put every description in the prompt, for every request.
• Let Claude look them up. The prompt lists only skill IDs and short names. Claude calls get_descriptors for the skills that might apply, up to 8 at a time.

This project uses lookup. The counts below come from Anthropic's token counting tool, measured once on the first request for Task A. The file stores only numbers.

counts = json.loads(Path("results/token-counts.json").read_text())
for name in ("demo", "xq"):
    c = counts[name]
    ratio = c["all_descriptors_in_prompt_tokens"] / c["lookup_design_tokens"]
    print(f"{name:5} {c['skills']:4} skills   lookup: {c['lookup_design_tokens']:6,} tokens   "
          f"all descriptors in prompt: {c['all_descriptors_in_prompt_tokens']:6,}   ({ratio:.1f}x)")

demo     8 skills   lookup:  3,546 tokens   all descriptors in prompt:  4,627   (1.3x)
xq     115 skills   lookup:  5,298 tokens   all descriptors in prompt: 51,468   (9.7x)

On the demo rubric the difference is small. On XQ, putting every description in the prompt makes each request about 10 times larger.

Caching narrows the cost difference. Opus 5.5 charges $0.20 per million tokens to reread cached text, instead of $4, so a large prompt that stays cached is cheap to send again. Two costs remain. The first request in each five-minute window pays extra to store the whole prompt. And every request fills Claude's context with 114 skills that do not apply, for the one that does. Lookup keeps each request small and shows Claude only what it needs.

Here is what Claude actually looked up in the recorded demo run:

from curriculum_auditor.replay import Recording

looked_up = {}
for entry in Recording.load(RECORDING).entries:
    for message in entry["request"]["messages"]:
        if message["role"] != "assistant":
            continue
        for block in message["content"]:
            if block.get("type") == "tool_use" and block["name"] == "get_descriptors":
                for skill_id in block["input"]["skill_ids"]:
                    looked_up[skill_id] = looked_up.get(skill_id, 0) + 1
print(looked_up)

{'DEMO.1.a': 6, 'DEMO.3.a': 3, 'DEMO.2.c': 6, 'DEMO.1.b': 4, 'DEMO.2.a': 3, 'DEMO.2.b': 5, 'DEMO.3.b': 2, 'DEMO.3.c': 2}

4. Claude judges, code verifies

Claude decides whether a section gives students a real chance to practice a skill. Code decides whether Claude's answer is complete and backed by the text. Every coverage submission needs:

• exactly one row for every skill, and
• a quote from the section for every skill marked supported, found word for word.

The cells below call the tools directly, without Claude, to show what gets rejected. Here strict_offsets=True also rejects wrong character positions for a quote. The recorded runs leave it off: if a quote is exact and appears only once, code fixes its position, because models are poor at counting characters.

task_a = lesson.section("measuring-heights-you-cannot-reach/task-a-the-ramp")
strict = Session(demo, lesson, responses, strict_offsets=True)

def rows(supported=None):
    supported = supported or {}
    out = []
    for skill_id in demo.ids():
        if skill_id in supported:
            text = supported[skill_id]
            start = task_a.text.find(text)
            out.append({"skill_id": skill_id, "status": "supported", "rationale": "The task asks for it.",
                        "evidence": [{"quote": text, "start": start, "end": start + len(text)}]})
        else:
            out.append({"skill_id": skill_id, "status": "not_evidenced", "evidence": [], "rationale": ""})
    return out

def attempt(label, submission):
    try:
        print(label, "->", strict.submit_section_coverage(task_a.id, submission))
    except ToolRejected as rejected:
        print(label, "-> rejected:", rejected.summary)
        for fix in rejected.problems:
            print("     ", fix)

fabricated = rows()
fabricated[3] = {"skill_id": "DEMO.2.b", "status": "supported", "rationale": "It asks for a ratio.",
                 "evidence": [{"quote": "Students compare sine and cosine graphs.", "start": 0, "end": 41}]}
attempt("Fabricated quote", fabricated)

missing = rows({"DEMO.2.b": "Choose sine, cosine, or tangent."})[:-1]
attempt("Missing skill", missing)

Fabricated quote -> rejected: Rejected. Fix every item below and resubmit the full row list. 2 attempt(s) left.
      DEMO.2.b quote 1: the quote does not appear in the section text. Copy it exactly, character for character. If the passage has symbols such as °, ², or ×, copy the symbol itself, or quote a shorter phrase that leaves the symbol out.
Missing skill -> rejected: Rejected. Fix every item below and resubmit the full row list. 1 attempt(s) left.
      Missing rows for 1 skill(s): DEMO.3.c. Every rubric skill needs one row, including not_evidenced.

Each rejection comes back as a list of fixes. After three rejections, the section is marked needs_review and the run moves on.

A section that never finishes cannot prove that a skill is missing. The skill might have been in that section. So the final report marks those skills needs_review, never not_evidenced.

attempt("Third bad try", missing)
attempt("Correct submission after three strikes", rows({"DEMO.2.b": "Choose sine, cosine, or tangent."}))

Third bad try -> rejected: Rejected 3 times. Section 'measuring-heights-you-cannot-reach/task-a-the-ramp' is now needs_review. Stop submitting it.
      Missing rows for 1 skill(s): DEMO.3.c. Every rubric skill needs one row, including not_evidenced.
Correct submission after three strikes -> rejected: Section 'measuring-heights-you-cannot-reach/task-a-the-ramp' is marked needs_review after 3 rejected submissions. Stop submitting it.

Scoring has the same checks and one more rule: evidence must come from the student's own words. Response r13 copied the task instructions into its answer. So a quote of that copied text is rejected, even though it appears in the student's work.

from curriculum_auditor.rollup import load_coverage_report

coverage = json.loads(Path("recordings/coverage-report-lesson-heights-demo.json").read_text())
scoring = Session(demo, lesson, responses)
load_coverage_report(scoring, coverage)

r13 = scoring.responses["r13"]
copied = "Measure the distance from your feet to the base with the tape measure."
start = r13.student_work.find(copied)
try:
    scoring.record_score("r13", "DEMO.3.a", "scored", 2, "DEMO.3.a.2",
                         [{"quote": copied, "start": start, "end": start + len(copied)}], "Lists steps.")
except ToolRejected as rejected:
    print(rejected.problems[0])

try:
    scoring.record_score("r13", "DEMO.3.a", "insufficient_evidence", 1, "DEMO.3.a.1", [], "No reasoning shown.")
except ToolRejected as rejected:
    print(rejected.problems[0])

quote 1: the student copied this text from the instructions. Copied task text is not evidence of the student's own work.
insufficient_evidence must have level null and descriptor_id null. Missing evidence is never Level 1.

4b. Your turn: four short exercises

You have seen the checks from the outside. Now write four of them yourself. Each one is a small function the project already relies on.

How each exercise works:

1. Read the task.
2. In the code cell, replace the raise NotImplementedError line with your code.
3. Run the check cell below it. It tests your function on several cases and tells you what passed, or which case to fix.

An unfinished exercise does not stop the rest of the notebook. If you are stuck, worked answers are in src/curriculum_auditor/exercise_solutions.py.

Exercise 1: Is the quote exact?

Claude gives a quote and the positions where it starts and ends. Write quote_is_exact so it returns True only when the text between start and end is exactly the quote.

A position past the end of the text should return False, not crash.

from curriculum_auditor.exercises import check

def quote_is_exact(source, quote, start, end):
    raise NotImplementedError("Write this function")

check(1, quote_is_exact)

Exercise 1: not started yet. Replace the line raise NotImplementedError(...) with your code, then run this cell again.

False

Exercise 2: Was it copied from the instructions?

A student can copy the task into their answer. That text is in their work, but it is not their thinking. Write is_copied so it returns True when the quote appears in the instructions and is at least min_chars characters long.

Why the length rule: a short quote such as "sine" can appear in the instructions by chance, and a student who writes it may mean it.

def is_copied(quote, instructions, min_chars=30):
    raise NotImplementedError("Write this function")

check(2, is_copied)

Exercise 2: not started yet. Replace the line raise NotImplementedError(...) with your code, then run this cell again.

False

Exercise 3: Is the score valid?

Write score_problems. It returns a list of what is wrong with one score, or an empty list if nothing is. The rules:

• status is "scored" or "insufficient_evidence".
• A "scored" result needs a level from 1 to 4 and at least one quote in evidence.
• An "insufficient_evidence" result must have no level. Missing evidence is never Level 1.

The check only looks at whether your list is empty. Write the problems in whatever words would help Claude fix them.

def score_problems(status, level, evidence):
    raise NotImplementedError("Write this function")

check(3, score_problems)

Exercise 3: not started yet. Replace the line raise NotImplementedError(...) with your code, then run this cell again.

False

Exercise 4: Is the absence earned?

A skill gets one mark from each section of the lesson: "supported", "needs_review", "not_evidenced", or None if that section never finished. Write skill_status to combine them into one final status:

1. If any section marked it "supported", it is "supported".
2. Otherwise, if any section marked it "needs_review" or never finished, it is "needs_review".
3. Otherwise, it is "not_evidenced".

Rule 2 is the important one. A section that never finished might have held the skill, so the tool cannot say the skill is missing.

def skill_status(section_marks):
    raise NotImplementedError("Write this function")

check(4, skill_status)

Exercise 4: not started yet. Replace the line raise NotImplementedError(...) with your code, then run this cell again.

False

5. The same agent with the Tool Runner

In section 2, you wrote the loop yourself: send the conversation to Claude, run the tools Claude asks for, send back the results, and repeat. The Anthropic SDK can run that loop for you. Its Tool Runner (client.beta.messages.tool_runner) repeats three steps until Claude stops asking for tools:

1. Send the conversation to Claude.
2. Run any tools Claude asks for.
3. Add the results to the conversation.

You give it your tools as Python functions. Wrapping a function with @beta_tool tells the SDK the tool's name, what it does, and what inputs it takes. curriculum_auditor/agent.py wraps the same five tools used in section 2.

Rejections work the same way as before. When a check fails, the tool raises the SDK's ToolError. The runner catches it and sends the list of fixes back to Claude, marked as an error, the same way the hand-written loop does.

The Tool Runner handles one conversation. Deciding what to work on next is still your code's job. The orchestrator in audit.py goes through the lesson one section at a time, and starts a new conversation for each section. That has two benefits:

• Each section gets Claude's full attention. A long section cannot fill up the conversation and crowd out the next one.
• A failure stays contained. If one section fails, the other sections are not affected.

from curriculum_auditor.audit import run_coverage

tool_runner_report = run_coverage(
    make_client("replay", recording="recordings/coverage-lesson-heights-demo-tool_runner.json"),
    Session(demo, lesson), runner="tool_runner", backend="replay", progress=lambda _: None)

print(tool_runner_report["summary"])
for skill in tool_runner_report["skills"]:
    where = ", ".join(s["section_id"].split("/")[-1] for s in skill["supported_in"])
    print(f"  {skill['skill_id']:9} {skill['status']:14} {where}")

{'supported': 5, 'not_evidenced': 1, 'needs_review': 2}

  DEMO.1.a  supported      task-a-the-ramp, task-b-something-tall-outside
  DEMO.1.b  needs_review   
  DEMO.2.a  supported      task-a-the-ramp, exit-ticket
  DEMO.2.b  supported      task-a-the-ramp, task-b-something-tall-outside, exit-ticket
  DEMO.2.c  supported      task-a-the-ramp, task-b-something-tall-outside
  DEMO.3.a  supported      task-a-the-ramp, task-b-something-tall-outside
  DEMO.3.b  not_evidenced  
  DEMO.3.c  needs_review   

The Tool Runner recording and the hand-written loop recording come from two separate live runs, so a few judgments differ. Replay repeats each run exactly. It does not make Claude give the same answer twice.

6. Scoring student work

Scoring works one student response at a time, with a new conversation for each. It scores only the skills that coverage marked supported for that response's task. For each skill, Claude:

1. Reads the four level descriptions.
2. Calls record_score with a level from 1 to 4 and quotes from the student's work, or
3. Marks it insufficient_evidence, with no level, when the work shows too little to judge.

Missing evidence is never scored as Level 1.

from collections import Counter
from curriculum_auditor.audit import run_scoring

scoring_session = Session(demo, lesson, responses)
load_coverage_report(scoring_session, coverage)   # hashes must match, or scoring refuses the coverage report
scores = run_scoring(make_client("replay", recording="recordings/scores-responses-demo-tool_runner.json"),
                     scoring_session, runner="tool_runner", backend="replay", coverage=coverage,
                     progress=lambda _: None)

print(scores["summary"])
print(Counter("IE" if row["level"] is None else row["level"] for row in scores["rows"]))
print()
for row in scores["rows"]:
    if row["response_id"] in ("r02", "r06") and row["skill_id"] == "DEMO.1.a":
        print(row["response_id"], row["skill_id"], "level", row["level"])
        print("   quote:", row["evidence"][0]["quote"][:100] if row["evidence"] else "(none)")
        print("   why:  ", row["rationale"])

{'scored': 60, 'insufficient_evidence': 12, 'missing': 0}
Counter({3: 21, 4: 18, 2: 15, 'IE': 12, 1: 6})

r02 DEMO.1.a level None
   quote: (none)
   why:   The student skipped Part 4 and wrote no check or estimate. That is missing work, not a Level 1 showing, so no level can be given.
r06 DEMO.1.a level 4
   quote: The two models only differ by about 1 mm.
   why:   Without being asked, the student checked the result against a second (tangent) model and explained why the two differ by 1 mm, which meets Level 4. The Pythagorean check reuses the same triangle, so on its own it proves little.

Comparing Claude with a teacher, on XQ

A teacher scored the 16 responses on four XQ skills, 32 scores in all, before Claude scored anything. Claude then scored the same responses. The report below compares the two, using only skill IDs and levels.

Read it as a demonstration of the method, not proof of accuracy. The responses were written by Claude for testing, and one teacher scored them. They show how to measure a scorer:

• Exact match: how often Claude and the teacher gave the same level.
• Within one level: how often they were at most one level apart.
• Mean absolute error: the average number of levels between them.
• Weighted kappa: agreement, adjusted for how often they would agree by chance.

"Insufficient evidence" is counted separately from the levels. Running Claude three times on the same work measures consistency: whether Claude gives the same answer again. That is not the same as being right.

from IPython.display import Markdown, display

reports = sorted(Path("results").glob("agreement-xq-*.md"))
if reports:
    display(Markdown(reports[-1].read_text()))
else:
    print("No agreement report yet. It is written by `curriculum-auditor agreement` after a live XQ run.")

Agreement report: XQ Competencies

Method demonstration on 16 synthetic responses scored by one rater. Agreement shows how closely Claude matched that rater, not whether the scorer is accurate on real student work. Consistency compares repeated runs of the same model and measures stability, not validity. No pass mark was set in advance.

Rubric: XQ Competencies (https://xqcompetencies.xqsuperschool.org/). Content hash 03bd012d4f1d. 32 labeled response-skill pairs.

• Run 1: claude-opus-5-5, effort high, 2026-09-30T03:57:01+00:00, about $3.04. Served by: {'claude-opus-5-5': 42}
• Run 2: claude-opus-5-5, effort high, 2026-09-30T04:10:59+00:00, about $2.84. Served by: {'claude-opus-5-5': 39}
• Run 3: claude-opus-5-5, effort high, 2026-09-30T04:25:25+00:00, about $2.92. Served by: {'claude-opus-5-5': 41}

Label change after model scores were seen: r13 was labeled 0 (no work to judge) in the blind labels. It was changed to 1 before any model scores existed, after a question about whether 0 was on the scale. It was restored to 0 after model scores existed, following a colleague's argument that a response with no work cannot be placed at any level. 0 is counted as insufficient evidence.

Claude against the human labels

Run	Exact, every paired case (IE as a category)	Exact, both gave a level	Within one level	Mean absolute error	Weighted kappa (linear)
1	65% (20/31)	86% (18/21)	100% (21/21)	0.14 (n = 21)	0.87 (n = 21)
2	69% (22/32)	87% (20/23)	100% (23/23)	0.13 (n = 23)	0.89 (n = 23)
3	74% (23/31)	91% (21/23)	100% (23/23)	0.09 (n = 23)	0.92 (n = 23)

Level statistics use only pairs where both gave a level. Insufficient evidence (IE) is counted apart:

• Claude said IE: 32% (10/31)
• The rater said IE: 6% (2/31)
• Missing from Claude's scores: 1 pair(s)
• Exact agreement with IE treated as a fifth category, over every paired case: 65% (20/31)

Scored or IE, run 1 (rows: rater, columns: Claude)

	Scored	IE
Scored	21	8
IE	0	2

Level confusion, run 1 (rows: rater, columns: Claude)

	1	2	3	4
1	7	1	0	0
2	0	5	1	0
3	0	0	4	1
4	0	0	0	2

Consistency across runs

Identical level in all 3 runs: 83% (25/30)

Runs	Exact	Within one level	Weighted kappa (linear)
1 vs 2	90% (19/21)	100% (21/21)	0.92 (n = 21)
1 vs 3	90% (18/20)	100% (20/20)	0.91 (n = 20)
2 vs 3	100% (21/21)	100% (21/21)	1.00 (n = 21)

Every pair

IE means insufficient evidence.

Response	Skill	Rater	Run 1	Run 2	Run 3
r01	FL.MST.1.e Checking results	3	3	3	3
r01	FL.MST.2.a Modeling	2	2	2	2
r02	FL.MST.1.e Checking results	1	1	1	1
r02	FL.MST.2.a Modeling	1	IE	IE	IE
r03	FL.MST.1.e Checking results	1	1	1	1
r03	FL.MST.2.a Modeling	1	1	1	1
r04	FL.MST.1.e Checking results	1	IE	IE	IE
r04	FL.MST.2.a Modeling	1	IE	IE	IE
r05	FL.MST.1.e Checking results	2	2	2	2
r05	FL.MST.2.a Modeling	1	IE	IE	1
r06	FL.MST.1.e Checking results	3	3	3	3
r06	FL.MST.2.a Modeling	3	4	4	missing
r07	FL.MST.1.e Checking results	1	2	2	2
r07	FL.MST.2.a Modeling	1	1	2	2
r08	FL.MST.1.e Checking results	1	IE	IE	1
r08	FL.MST.2.a Modeling	1	IE	1	IE
r09	FL.ID.3.b Explaining my reasoning	3	3	3	3
r09	FL.MST.2.c Explaining models	4	4	4	4
r10	FL.ID.3.b Explaining my reasoning	1	IE	IE	IE
r10	FL.MST.2.c Explaining models	1	IE	IE	IE
r11	FL.ID.3.b Explaining my reasoning	2	2	2	2
r11	FL.MST.2.c Explaining models	2	2	2	2
r12	FL.ID.3.b Explaining my reasoning	3	3	3	3
r12	FL.MST.2.c Explaining models	1	1	1	1
r13	FL.ID.3.b Explaining my reasoning	IE	IE	IE	IE
r13	FL.MST.2.c Explaining models	IE	IE	IE	IE
r14	FL.ID.3.b Explaining my reasoning	2	3	2	2
r14	FL.MST.2.c Explaining models	4	missing	4	4
r15	FL.ID.3.b Explaining my reasoning	4	4	4	4
r15	FL.MST.2.c Explaining models	2	2	2	2
r16	FL.ID.3.b Explaining my reasoning	1	1	1	1
r16	FL.MST.2.c Explaining models	1	1	1	1

7. The same tools over MCP

MCP (Model Context Protocol) lets an AI app, such as Claude Code, use outside tools. curriculum_auditor/server.py offers the same five tools over MCP, plus three more: load_curriculum, get_section, and coverage_report. No API key is needed, because the app itself is the model. To add it to Claude Code, run this from the project folder:

claude mcp add curriculum-auditor -- uv run curriculum-auditor-mcp

The cell below starts the server and talks to it the way Claude Code would.

import sys
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client

params = StdioServerParameters(command=sys.executable, args=["-m", "curriculum_auditor.server"], cwd=str(ROOT),
                               env={**os.environ, "CURRICULUM_AUDITOR_ROOT": str(ROOT)})

async with stdio_client(params) as (reader, writer):
    async with ClientSession(reader, writer) as mcp_session:
        await mcp_session.initialize()
        print(sorted(t.name for t in (await mcp_session.list_tools()).tools))
        loaded = await mcp_session.call_tool("load_curriculum", {"curriculum_path": "content/lesson-heights.md",
                                                                 "rubric": "demo"})
        print(loaded.structured_content["skill_count"], "skills,",
              len(loaded.structured_content["sections"]), "sections")
        check = await mcp_session.call_tool("check_answer", {"item": {
            "id": "B4", "mode": "numeric", "expected": "15*tan(38) + 1.5", "actual": "6.15", "unit": "m",
            "actual_unit": "m", "angle_convention": "degrees", "absolute_tolerance": "0.005"}})
        print("B4:", check.structured_content["status"])

['check_answer', 'coverage_report', 'get_descriptors', 'get_section', 'list_competencies', 'load_curriculum', 'record_score', 'submit_section_coverage']

8 skills, 7 sections
B4: incorrect

8. Optional: a live run on your own lesson

Everything above replayed saved recordings. To call Claude for real:

1. Copy .env.example to .env and add your ANTHROPIC_API_KEY.
2. Download XQ: uv run curriculum-auditor fetch.
3. In the cell below, set RUN_LIVE = True and point LESSON at any Markdown lesson.

A live coverage run of this lesson against XQ cost about $1.50 in testing. The run stops if spending passes the budget you set. Results go to the reports/ folder, which is never saved to the repository.

RUN_LIVE = False
LESSON = "content/lesson-heights.md"
BUDGET_USD = 3.0

if RUN_LIVE:
    from curriculum_auditor.client import Meter
    from curriculum_auditor.reports import write_report
    live_report = run_coverage(make_client("live"), Session(load_rubric("xq"), load_curriculum(LESSON)),
                               runner="tool_runner", backend="live", meter=Meter(BUDGET_USD))
    print(write_report(live_report, "reports", "coverage-live"))
else:
    print("Skipped. Set RUN_LIVE = True to call the API.")

Skipped. Set RUN_LIVE = True to call the API.