Public benchmark · v1 · opened 2026-08
Sanskrit AI Agent Benchmark
One question, scored: can a coding agent turn a thousand irregular Sanskrit books into a structured, self-consistent, queryable corpus without a human in the loop?
Hero
The challenge. Given 794 raw HTML pages of Sanskrit text — inconsistent encodings, four transliteration schemes, no shared markup convention, no table of contents — produce a single validated corpus artifact that a search engine could serve, in 35 minutes, under 512 MiB.
- Source pages
- 794
- Time limit
- 35 min wall clock
- Artifact cap
- 512 MiB
- Submissions
- 5
- Best reward
- 0.612
- Status
- open
Task definition
Corpus. 794 source pages drawn from the GRETIL snapshot, spanning Vedic saṃhitās, epics, purāṇas, śāstra and commentary literature. Pages vary in encoding (UTF-8 Devanāgarī, IAST, Harvard-Kyoto, SLP1), in markup depth, and in whether verse boundaries are marked at all. No page-level schema is supplied.
Required output. A build directory containing manifest.json, an executable build entrypoint, and JSONL record streams for documents, passages, dictionary entries, analyses and occurrences. Every record carries a stable id; every foreign key must resolve within the artifact.
JSONL record schema (passages.jsonl)
| Field | Type | Required | Note |
|---|---|---|---|
| id | string | yes | stable, globally unique |
| document_id | string | yes | must resolve in documents.jsonl |
| position | integer | yes | monotonic within document |
| text_deva | string | one of | Devanāgarī surface form |
| text_iast | string | one of | IAST surface form |
| ref | string | no | canonical citation, e.g. bg.2.16 |
| source | object | yes | {file, offset} into the input snapshot |
Constraints. No network access after the dataset is extracted. No human intervention. 35 minute agent wall clock including build time. Final artifact must be 512 MiB or smaller — this is a hard gate, not a penalty term.
Scoreboard
| Harness | Model | Reward | Status | Date | Submission |
|---|---|---|---|---|---|
| toroid | deepseek/deepseek-v4-flash-0731 | 0.000 | failed | 2026-08-18 | view run |
| toroid | anthropic/claude-opus-5 | 0.612 | partial | 2026-08-14 | view run |
| swe-shell | openai/gpt-5.2-codex | 0.548 | partial | 2026-08-11 | view run |
| toroid | google/gemini-3.5-pro | 0.331 | partial | 2026-08-09 | view run |
| minishell | qwen/qwen4-coder-480b | 0.000 | failed | 2026-08-06 | view run |
Reward is the weighted sum in section 04, multiplied by the within_512_mib gate. A failed gate zeroes the run regardless of corpus quality.
Methodology
| Metric | Weight | What it measures |
|---|---|---|
| submission_contract | 0.10 | Required files present, entrypoint exits 0 |
| manifest_valid | 0.05 | manifest.json parses and declares every stream |
| build_executable | 0.05 | build runs from a clean checkout |
| jsonl_valid | 0.08 | Every line parses as a JSON object |
| schema | 0.15 | Records satisfy the declared record schema |
| source_coverage | 0.18 | Fraction of the 794 source pages represented |
| analysis_quality | 0.18 | Morphology and lemma assignment sampled against gold |
| referential_integrity | 0.13 | Fraction of foreign keys that resolve |
| compactness | 0.08 | Bytes per passage relative to a reference encoder |
| within_512_mib | gate | Multiplies the total by 1 or 0. No partial credit. |
Weights sum to 1.00 before the gate is applied. Gold samples for analysis_quality are held out and rotated between benchmark versions.
Reproduce it
- Size
- 318 MiB compressed
- SHA-256
- 4f9c1ab2e7d0…8b31
curl -O https://data.yashbonde.com/artha-bench/v1/yashbonde-artha-bench.tar.zst
sha256sum yashbonde-artha-bench.tar.zst
zstd -dc --long=31 yashbonde-artha-bench.tar.zst | tar -xf -
git clone https://github.com/yashbonde/artha-bench && cd artha-bench
python -m artha_bench.run --dataset ../artha-bench/v1 \
--harness toroid --model <provider/model> --limit-minutes 35
python -m artha_bench.score --submission runs/latest
Extraction requires zstd -dc --long=31: the archive was written with a 2 GB window, and a default-window decoder fails with "Frame requires too much memory for decoding".
Results analysis
The 2026-08-18 toroid / deepseek-v4-flash-0731 run is the most informative failure so far. On corpus content it did well: 359 documents, 170,488 passages, 1,101,446 dictionary entries, 541,021 analyses and 4,054,517 occurrences, with referential integrity of 1.0 — every foreign key across five streams resolved. That is a genuinely coherent artifact.
It scored 0.000 anyway. The finished artifact came to 1.17 GB against a 512 MiB cap, and within_512_mib is a multiplicative gate, so the entire weighted sum was zeroed. The agent never traded fidelity for size — it emitted full per-occurrence records with repeated surface strings rather than interning them, and nothing in its plan checked artifact size until the end, by which point there was no time to re-encode.
It also hit the 35 minute agent timeout before writing manifest.json, which independently zeroes manifest_contract and submission_contract. So there were two fatal problems, both of them schedule problems rather than Sanskrit problems.
The pattern across submissions is consistent: models are competent at the linguistic work and weak at budgeting. The two passing-quality runs both wrote a size check into the build loop early; the two zeroed runs both discovered their budget at the end.
Passages
170,488
Occurrences
4,054,517
Artifact size
1.17 GB
Submit
- Open a submission PRAdd a run directory under
submissions/with your scorer output and harness config - Read the rules in fullWhat counts as human intervention, and what re-runs are permitted
- Email a private runFor unreleased models; results are published, weights and prompts are not
Every submission must include the full agent transcript and the scorer's JSON output. Runs that cannot be re-scored from the published dataset are listed but not ranked.