19 August 2026: Maps, mirrors, and missing links

Project Artha needs data that a machine can query and a scholar can trace. The GRETIL source material does not have one stable shape. It contains verse, prose, commentary, headings, notes, and many local conventions. We used this material to test whether a coding agent could inspect an unknown corpus, write a parser, run it at full scale, and leave evidence that the result was complete.

The first complete corpus builds

OpenCode, with DeepSeek V4 Flash, completed the corpus structure task. It produced 987,673 records and processed 167,064,387 normalized characters. The public verifier accepted the submission. The run took 1 hour, 11 minutes, and 47 seconds.

Pi later completed the same task with the same model. It produced 1,085,459 records and processed 175,839,598 normalized characters. The verifier also accepted this submission. Pi finished in 34 minutes and 46 seconds.

These two runs showed that a general coding harness could inspect the source, create specialized parsing code, and produce a large queryable corpus without a human writing the parser first.

The same model did not produce the same result

We ran DeepSeek V4 Flash through several harnesses. Toroid produced 1,384,473 units and Codex produced 1,322,088 units, but both failed the submission contract. Claude Code stopped before it produced corpus output. DeepSeek Harness ended after its task disk filled.

Model choice did not explain the whole outcome. The harness changed how the model inspected files, managed long commands, used disk, and decided when the task was complete. A strong model could still produce an invalid artifact when its execution loop handled the work badly.

Nemotron produced two accepted corpus artifacts

OpenCode, with Nemotron 3 Ultra 550B A55B, completed the corpus task in 43 minutes and 22 seconds. It produced 937,642 records, processed 140,928,712 normalized characters, and passed the verifier.

Claude Code, with the same model, produced a contract-valid artifact before its final response stream failed. The artifact contained 920,775 records and 133,617,465 normalized characters. The work finished in 26 minutes and 52 seconds. We count the artifact as complete because the required files existed and the verifier accepted them.

Toroid stopped after 27 seconds. DeepSeek Harness lost its response stream. Codex lost its stream after 18 minutes and 44 seconds. Pi reached an idle timeout after 18 minutes and 48 seconds. These runs completed as infrastructure results, but they did not produce corpus artifacts.

Gemma could not enter the benchmark

We sent the same task to OpenCode, Toroid, DeepSeek Harness, Claude Code, Codex, and Pi with the free Gemma 4 31B IT endpoint. The shared provider rejected every run with rate-limit responses. The experiment finished with a clear infrastructure result. It did not measure the model’s ability to solve the task.

A high score exposed a weak verifier

We created a second benchmark that asked agents to reduce 20 GRETIL dictionaries into one reusable SQLite lexicon. Toroid finished the first run in 19 minutes and 28 seconds. It created 931,651 entries, 1,208,991 sense rows, and 286,318 lexical nodes. The first verifier gave the artifact a score of 0.986.

A direct review found a serious defect. Every sense row had a null entry link, and the forms table was empty. The database looked complete by row count, but later tasks could not join senses back to entries. The experiment showed that structural counts alone do not prove data quality.

We replaced the weak checks with hard relational checks. The new verifier requires linked senses, useful definitions, extracted forms, exact source reconciliation, valid memberships, non-null relationship columns, and passing SQLite integrity checks. The old artifact now scores zero under this verifier.

Twenty sources did not mean the same twenty sources

The unlimited runs passed the stricter relational checks, but a direct comparison found another problem. Each artifact contained 20 sources, yet the agents did not choose the same sources.

Toroid processed GRETIL rows 802 through 821. This set included Bloomfield’s Vedic Concordance but excluded Mani’s Puranic Encyclopaedia. Pi and OpenCode included the Amarakośa and rows 803 through 821, but excluded both Bloomfield and Mani. DeepSeek Harness processed rows 803 through 822. It included Mani but excluded Bloomfield and the Amarakośa.

Entry 6 in the Toroid artifact made the problem easy to see. Toroid stored aṃśas as a headword and stored “aṃśas te hastam agrabhīt” with its textual references as a meaning. The source is a concordance, not a normal dictionary. The line records where a Vedic phrase occurs; it does not define the word. Pi, OpenCode, and DeepSeek Harness have no matching aṃśas entry because they did not process Bloomfield’s concordance.

All four artifacts contain the related dictionary headword aṃśa, but their coverage differs. Toroid has 41 entries from 16 sources. Pi has 19 entries from 15 sources. OpenCode has 42 entries from 15 sources. DeepSeek Harness has 19 entries from 15 sources. Their sense counts also differ.

The verifier checked the number of sources, not their exact identities. Therefore, the four scores are not fully comparable. Future runs need a fixed source manifest with exact GRETIL row IDs and checksums. The task must also distinguish dictionaries from concordances, encyclopaedias, and other lexicographic sources.

Verification became part of the benchmark

The completed experiments changed the task design. A benchmark must test the properties that later work depends on. File presence, row count, and clean SQL are not enough. The verifier must test links, coverage, reconstruction, and useful queries.

The corpus experiments produced several accepted artifacts and a clear harness comparison. The lexicon experiment found a false success and corrected the evaluation contract. Together, these results give Project Artha a stronger base for the next benchmark.