Structuring 794 Sanskrit books: what one agent run revealed

GRETIL is the best free archive of machine-readable Sanskrit and it is also a museum of thirty years of transcription conventions. The 794 pages we pull from it share almost nothing structurally: verse numbers appear as || 2.16 ||, [16], 16. and bare right-aligned digits; some files interleave commentary with root text without marking the switch; six different encodings appear, three of them undeclared.

Structuring that is not a parsing problem with one right answer. It is several hundred small editorial decisions — where a chapter starts, whether a colophon is a passage, what to do when an edition renumbers — each of which has to be made consistently or the resulting IDs are worthless.

That is exactly the shape of task we built the benchmark around: long-horizon, weakly specified, verifiable only at the end. The question is not "can a model read Sanskrit" but "can it hold one convention across 794 files and ship the result in a form someone else can load".

The run lasted just under eleven hours of wall clock. The agent chose a sensible architecture unprompted: a normalising pass to UTF-8 NFC, a per-file convention detector, then a streaming writer emitting one JSONL file per record type. It wrote its own regression check that re-read every emitted passage and confirmed the ID resolved back to a byte offset in the source file — a check we had not asked for.

The record schema it settled on:

{"type":"passage",
 "id":"bg.2.2.16",
 "doc_id":"bg",
 "section":"2",
 "ordinal":612,
 "text_dev":"नासतो विद्यते भावो …",
 "text_iast":"nāsato vidyate bhāvo …",
 "src":{"file":"sa_bhagavadgita.htm","offset":184213,"len":118}}

{"type":"occurrence",
 "id":"occ.bg.2.2.16.3",
 "passage_id":"bg.2.2.16",
 "surface":"विद्यते",
 "lemma":"vid",
 "analysis_id":"an.vid.pres3sg.pass",
 "dict_refs":["mw.0963.2"]}

Every passage_id, analysis_id and dict_ref in the submission resolves — 4,054,517 occurrences pointing into 541,021 analyses and 1,101,446 dictionary entries with zero dangling keys. That is the hard part of this task and the agent did it.

What it did not do: keep the output under the 512 MiB budget (the uncompressed drop was 1.9 GiB, largely because text_iast was duplicated onto every occurrence), emit a valid manifest.json, or leave behind a build command that ran. The submission contract asked for all three in the task statement. Two of the three failures are one-line fixes; the size overrun is a schema decision made in hour one and never revisited.

Because the harness scores the delivered artefact rather than the intermediate work, a run that got the linguistics substantially right scored the same as one that produced nothing. That is deliberate, and it is the finding: agents on long tasks optimise the part they can see themselves succeeding at, and drift off the part that only gets checked at the end.

One run. n=1. We have no variance estimate; a second run could plausibly clear the manifest gate by accident and score 0.6.

One model, one harness. These numbers describe a single model under a single scaffold with a single tool set. They are not a statement about agents in general, and cross-model comparison on this task is not yet meaningful.

Timeout confound. The run hit the wall-clock ceiling at 359 of 794 works. The 0.5 source_coverage score therefore measures throughput, not capability, and the missing manifest may simply be work that was scheduled last and never reached.

Verifier immaturity. analysis_quality is sampled against roughly 2,000 hand-annotated tokens. That is enough to rank, not enough to trust a difference of a few points.