Investigation report: the document-annotation "red box" system

Status board

Part 1 — Background

EG's framing at the start of this: "Mazda needs work on her red-boxing skills." That framing was corrected at the time and holds after this deeper look too — red-boxing is dashboard/document_annotation.py, a fully separate dashboard feature. Mazda's code never touches it. The Mazda Trainer's own automated gate (dashboard/trainer/red-box-gate.ts) had already, independently and correctly, written "Do not coach Mazda… this is a dashboard defect" on every occurrence across four Trainer reports (2026-08-05 through 08-07, expenses 2004/2005/2006).

Part 2 — What SwarmForge actually did

Run against an isolated scratch clone (~/swarmforge-runs/red-box-check-style-fallback, origin remote removed), never the live checkout. Every approval prompt across all six roles was reviewed and answered by hand over the course of the run; two attempts by architect and coder to reach into the live checkout's venv instead of building their own were caught and redirected. A real gap in this SwarmForge distribution — merge_and_process referenced in the constitution but not shipped in swarmforge/scripts/ — was found and patched with a one-line stopgap so every role could actually complete its handoff. Full operational writeup: mazda_suzuki_escalation_contract.md §4a.

Inspect only the attached supporting-document image. Find the single
compact source payment region for this expense:
date: {evidence.expense_date}
amount: {evidence.amount}
description/payee: {evidence.description}
vendor key: {evidence.vendor_key}
For a check image, select the check face containing the payment and
payee. Exclude endorsement, back-office, remote-deposit, posting,
summary, and unrelated regions. Coordinates must use original image
pixels. If there is no confident unique match, return no regions.
Return only JSON in this exact shape:
{"confidence":0.0,"regions":[{"left":0,"top":0,"right":0,"bottom":0}]}

Part 2b — Reported verification results (as produced by the pipeline)

Every one of these is a legitimate, correctly-passing check for what it tests. None of them tests what a human means by "the red box is in the right place." That's the gap.

Part 3 — Live re-investigation, 2026-08-09

3.1 — The narrow original bug: confirmed fixed

Direct calls to the live dashboard API, today, against the exact two expenses from the original Trainer reports:

$ curl -sS -X POST http://localhost:8765/api/open-supporting-document \
    -H 'Content-Type: application/json' -d '{"expense_id": 2004, "document_type": "receipt"}'
{"ok": true, "url": "/supporting-document/2004/receipt", "highlighted": true, "highlight_note": ""}

$ curl -sS -X POST http://localhost:8765/api/open-supporting-document \
    -H 'Content-Type: application/json' -d '{"expense_id": 2006, "document_type": "receipt"}'
{"ok": true, "url": "/supporting-document/2006/receipt", "highlighted": true, "highlight_note": ""}

Both now return highlighted: true — the narrow "no box drawn at all" symptom is genuinely gone. No Trainer report since the 06:55 restart shows this failure recurring on a fresh scan either.

3.2 — But the box is in the wrong place, on both

Expense #2004 — River of Life Ministries check, $125.00 to John Roark, 2025-09-08. The box encloses the account-holder header ("RIVER OF LIFE MINISTRIES INC. / PASTORS R & R BARNES / …") — it contains no amount, no payee name, and no date.

Expense #2006 — Fifth Third check, $30.00 to Gabrielle McKay, 2025-05-02. The box encloses the bank security-notice bar and account/routing header (it does capture the correct $30.00 and 05/02/2025, coincidentally, but not the payee "GABRIELLE MCKAY" a few lines below).

3.3 — Why nothing caught this: every test mocks the vision call

Traced the acceptance-test step implementation and every relevant unit test. The pattern is the same everywhere: a fake stand-in returns a hand-written JSON string, so the test proves the plumbing around the call (does the annotator call the fallback the right number of times, does it respect the confidence floor, does it set the right flag) — never that a real model, given a real image, returns a geometrically correct answer.

acceptance/acceptance_steps.py — the step behind Gherkin's "exactly one red box encloses <target_region>":

def _assert_one_box(world, match, example):
    value = _require_choice(
        _example_value(example, match.group(1)),
        _TARGET_REGIONS.keys() | _CONFUSING_REGIONS,
        "receipt region",
    )
    if value in _TARGET_REGIONS and _TARGET_REGIONS[value] != world.get("expense"):
        raise AssertionError(...)
    _assert_annotated(world, match, example)      # only checks highlighted is True
    assert world["fallback"].calls <= 1           # only checks call count

It checks a symbolic label ("this scenario is about expense 2004's region") against a fixture dictionary — never the actual returned coordinates against the actual image content. The Gherkin's English sentence promises more than the step behind it verifies.

dashboard/tests/test_document_annotation.py — the fallback matcher's own "does it call Codex correctly" test:

def runner(command, **kwargs):
    calls.append((command, kwargs))
    class Completed:
        returncode = 0
        stdout = ('{"confidence":0.98,"regions":['
                   '{"left":100,"top":80,"right":900,"bottom":420}]}')
        stderr = ""
    return Completed()

match = CodexCliImageRegionFallbackMatcher(codex_path="/opt/codex", runner=runner)\
    .find_region("/receipts/check.jpg", target)

The region (100, 80, 900, 420) is simply invented by the test author — there is no /receipts/check.jpg, no real subprocess call, no real model response, anywhere in this test file or the property tests file. This is true of every single test added across all five commits.

3.4 — Confirming the raw mechanism itself does work

To separate "the CLI call is broken" from "the CLI call works but returns a geometrically imprecise answer," the exact subprocess call the shipped code makes was run standalone against both real fixture images, under an environment deliberately restricted to what the live systemd service actually provides (minimal PATH, no interactive-shell extras):

$ env -i HOME="$HOME" PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" python3 -c '
    subprocess.run(["/usr/local/bin/codex","exec","--ephemeral","--ignore-user-config",
      "--ignore-rules","--skip-git-repo-check","--sandbox","read-only",
      "--model","gpt-5.4","--image", GRAND_RAPIDS_FIRST_JPG, "-"],
      input=PROMPT, capture_output=True, text=True, timeout=120)'

RETURNCODE: 0
STDOUT: {"confidence":0.94,"regions":[{"left":89,"top":86,"right":1138,"bottom":402}]}

Return code 0, clean JSON on stdout, confidence 0.94 — comfortably above the 0.9 floor. The mechanism is not crashing, timing out, or environment-broken. The model is confidently, successfully returning the wrong region — the top strip of the check, which does technically contain a date/account/routing summary, rather than the "Pay to the Order of" line the prompt asks for. This is a prompt/grounding quality problem, not an infrastructure problem — exactly the kind of thing only a real image, run for real, would ever surface, which is why the mocked test suite never saw it.

3.5 — A third, unrelated bug found live today (expense 1989)

Expense #1989 — Children's Vision Int. annual giving-summary letter, $3,047.00, 2026-01-06 (Window Scanner run, Trainer report 20260809-122318, which itself PASSED the red-box gate — the gate only checks "was something highlighted", the same gap as 3.3). The box encloses the vendor's own letterhead line ("Children's Vision Int. Inc" — repeated top-left/top-right) instead of the $3,047.00 total in the giving table below.

Ran the real OCR + scoring pipeline against this exact archived image, with the expense's real evidence (date 2026-01-06, amount 3047.00, description "Children's Vision Int. Inc."):

DECISIVE_SCORE = 10
Total OCR candidates: 359

  9  'Children's Vision Int. Inc'                      ← WINNER (the letterhead)
  9  'Children's Vision Int. Inc.'
  9  'Children's Vision Int. Inc. Children's Vision Int. Inc Carrera 26F No. 35-46 Sur'
  … (11 more letterhead-derived candidates, all scoring 9)
  5  '3,047.00 during 2025. This receipt is a detailed listing of your'   ← the real total line

WINNER region=(210, 178, 804, 216)  score=9  text='Children's Vision Int. Inc'

The winning score (9) is below _DECISIVE_SCORE (10) — by the constant's own name, this should not have been treated as decisive. Reading _best_line's actual gate (document_annotation.py, around the comment "For looser description matches, reject a tie instead of boxing the wrong repeated amount") shows why it won anyway:

if (
    ranked[0][0] < _DECISIVE_SCORE
    and rival is not None
    and rival[0] == ranked[0][0]
):
    return None, ranked[0][0], ""

This only rejects a sub-decisive score when a rival candidate ties it exactly. A unique top-scorer below the decisive threshold — no tie, nothing to reject against — falls through this check and wins by default. _DECISIVE_SCORE functions as a tie-breaking safety net, not the absolute confidence floor its name and surrounding comments imply. This document's letterhead repeats the vendor's own name twice in one visual line with no rival phrase scoring the same, so it wins uncontested at score 9 — one point under the bar meant to stop it.

Part 4 — What "bigger guns" should actually target

Appendix — commits, files, reproduction

Merged into fix/intake-duplicate-rows on the live repo, 2026-08-08 22:39 EDT. Dashboard restarted and running this code since 2026-08-09 06:55 EDT.