Methods
How tasks are built, graded, checked and calibrated. Calibration figures cover the research-backtest family only.
01Scope
A task is a research backtest built from raw synthetic market data that contains undisclosed data-quality traps. The agent receives the data and a strategy spec and submits /app/solution.py.
Strategy families currently calibrated, all monthly decile long-short (tasks in the 50-task private set):
02Data
03Trap categories
A correct solution must handle each category present in the data.
04Grading
Grading is deterministic and sandboxed. The submitted code is re-run on hidden datasets from 5 seeds. Holdings are checked per trap category, plus a clean-data sanity check. Pass = 100/100. The reward is binary at 100; a continuous score is also reported.
Determinism check: 36 of 36 saved responses re-graded identically (Oct 14).
05Acceptance (grader soundness)
Every task passes these checks before it is scored or shipped:
06Tamper hardening
The verifier runs in a separate container. Only /app/solution.py crosses over. Submissions run as an unprivileged user, and the process is stopped before the reward is written.
Harbor test on an earlier sample task (Oct 2026): the oracle scored 1. No-op, stub, and attempts to write the reward file, read the grader or keep running after exit all scored 0. The full per-task Harbor tamper run is in progress.
07Calibration protocol
08Results
Private set (50 tasks, Oct 14–19, 2026), aggregates only. Task names withheld.
Pooled 290/900 (32.2%, 95% CI 29–35%). 49 of 50 tasks in the 2–70% band; 1 at 72%, labelled easy; none at 0%.
free sample pack (3 tasks, separate from the 50-task set, 6 attempts per model): easy, 15/18 (83%) · in band, 7/18 (39%) · hard, 0/18
09Limitations
- Small n per task: 6 attempts per task per model (18 pooled), so about ±20 points on a task's pooled rate and up to about ±30 for one model on one task. Per-model totals over 300 attempts are about ±5 points.
- The 40 tasks added Oct 19 were picked for the band using their first 4 attempts per model, and those attempts are included in the 6, so band placement partly reflects selection.
- Single-response mode only. Agentic pass rates have not been measured.
- One configuration per model.
- One task design pattern (monthly decile long-short research backtest), across seven strategy families.
- No human blind solves yet.
10Data-quality benchmark
The data-quality benchmark tests whether a model can read a messy market-data CSV and flag duplicates, gaps, bad prices, and unadjusted splits. Each task is procedurally generated: a seeded random walk produces clean prices, then known problems are planted at recorded rows, which becomes the answer key. The model returns its findings, and a deterministic scorer outside the model checks them against the key.
The benchmark was built and demoed during Meta's Global AI Developer Hackathon (Oct–Nov 2026) to test Meta's Muse models on hard quant-finance data-cleaning work. Results are on the leaderboard. Source: github.com/alsopgroup/lanemint-meta-hackathon · Dataset: LaneMint/lanemint-meta-benchmark
11Public vs private sets
The private, sellable set (50 tasks as of Oct 19, 2026) produces the calibration aggregates. The public held-out set produces the leaderboard. Public set v1 was scored Oct 15, 2026. The public set carries canary strings. If a public task leaks, it is rotated out, replaced with a fresh variant from the generator, and the rotation is logged.
12Delivery format
harbor/<task>/
task.toml # schema 1.4, separate verifier mode, limits, canary
instruction.md # spec + methodology
environment/ # Dockerfile (python 3.11, numpy, pandas, pyarrow)
# + sample data
Harbor schema 1.4. Pinned environment: Python 3.11, numpy, pandas, pyarrow. The free sample pack is graded by LaneMint. Verifier-included packages are available under NDA on fresh variants.
13Changelog and contact
Changes to the grader, harness and sets are logged on the leaderboard changelog. Questions: hi@lanemint.com.