An autonomous agent ran the machine-learning research loop — measure, diagnose, invent, implement, verify, decide — across 21 runs with zero manual interventions. This is every experiment it ran, including the ones that failed and the guards that caught them.
The validation-best checkpoint at convergence, measured over 20 seeds, then scored once on the hidden test split after the configuration was frozen. Test was unreachable during the search, so no selection decision could be contaminated by it.
| split | GAUC | nDCG@5 | primary | baseline primary | score_dataset |
|---|---|---|---|---|---|
| validation, 20-seed mean | 0.6702 | 0.5373 | 0.6038 | 0.6016 | +0.0022 |
| hidden test, scored once | 0.6639 | 0.5304 | 0.5972 | 0.5946 | +0.0026 |
The gain held on test and grew slightly. Validation sd over 20 seeds is 0.0003 and all 20 seeds beat the baseline. The run converged in 5 research cycles of the 50 permitted, logging 13 experiments once its 8 exploit trials are counted, on 105,304 tokens and 59 minutes of wall-clock with zero manual interventions.
One cell per experiment, ordered as it happened. Most were caught before they cost a training run: a patch that reads as the intervention and changes nothing is the most common failure here, and it is detected in 0.4 seconds.
The official baseline is 0.6016 and the oracle ceiling is 0.8484: 27.1% of evaluation users have no positive label, so their nDCG is zero for any model. Total headroom is 0.247, and the organisers’ own ablations move this metric by less than 0.002. Everything below sits inside a 0.004 window.
The best of these measures 0.6038 over 20 seeds (sd 0.0003), which matches the human-tuned result exactly — by a different route. The human path was a listwise objective trained on evaluation-length lists plus a retuned learning rate; the agent reached the same score through curriculum learning over sequence length and a length-normalisation feature. It did not rediscover the human path.
An exploit trial measured a clear improvement on the first seed. Run on five seeds, it averaged to almost nothing. The keep/revert decision is arithmetic, not a judgement call, and it declined.
The gate was right. A human looked at the same numbers and spent an hour chasing it.
The same rule applied in the other direction on the final run. An accepted result read 0.6040 on one seed, and a neighbouring configuration in the same sweep read 0.6040 and measured 0.6034 across five. So it was tested at 20 seeds before anything was claimed: mean 0.6038, sd 0.0003, every seed above the baseline. The caution was right to apply, and this time the result survived it.
The scoring function is a factorisation machine throughout, the same architecture as the official baseline. Every accepted improvement came from the training objective or the input features. 35 experiments edited the model and none of them was accepted.
| Module | Experiments | Share | What it controls |
|---|---|---|---|
| train.py | 180 | 51% | objective, optimiser, batching |
| features.py | 135 | 39% | what the model sees |
| model.py | 35 | 10% | the scoring function itself |
| Primary | Module | Technique |
|---|---|---|
| 0.6040 | features.py | length-normalization |
| 0.6031 | train.py | weighted-loss-imbalance |
| 0.6031 | train.py | class-weighted-loss |
| 0.6030 | train.py | focal-loss-weighting |
| 0.6028 | features.py | temporal-patterns |
| 0.6023 | train.py | curriculum-learning |
| 0.6022 | features.py | temporal-decay-weights |
| 0.6022 | features.py | temporal-decay-relevance |
| 0.6022 | features.py | temporal-activity-features |
| 0.6014 | features.py | user-cluster-features |
The human improvement came from training the ranking objective on lists the length of the evaluation lists. The agent reached for that same family — listwise objectives, group sizing, list-length handling — in 50 experiments across the project, and 0 were accepted. It repeatedly identified the right mechanism and never made it work, while every win it did land came from somewhere else.
Diagnosis was never the bottleneck. Selection and implementation were.
Each generation added a stage in response to a measured failure of the last. Two were built, measured, and reverted.
| Gen | Runs | Improved | Experiments | Best |
|---|---|---|---|---|
| V1 | 1 | 0 | 20 | — |
| V2 | 1 | 0 | 25 | — |
| V3 | 1 | 0 | 25 | — |
| V4 | 4 | 3 | 61 | 0.6031 |
| V5 | 4 | 2 | 68 | 0.6022 |
| V6 | 6 | 4 | 110 | 0.6040 |
| V7 | 2 | 0 | 48 | — |
| V8 | 2 | 0 | 30 | — |
Five roles, each on the model it needs. The judge writes only prose because its verdict is arithmetic; the coder is the measured bottleneck and the largest single cost.
| Role | Tokens | Share | Job |
|---|---|---|---|
| coder | 1752K | 47% | write correct numpy |
| inventor | 913K | 24% | propose interventions |
| classifier | 723K | 19% | name the problem |
| analyst | 295K | 8% | pick tools, report numbers |
| reflector | 51K | 1% | explain a decision already made |
4.9 GPU-hours across 10.1 hours of wall clock, on a laptop CPU with no accelerator. numpy and pandas only.
Including the architecture generations that were built, measured, and reverted because they did not produce a win.
| Run | Arch | Exps | Contract sat. | Keeps | Best | Tokens | Stopped |
|---|---|---|---|---|---|---|---|
| 01-library-pipeline-20cyc | V1 | 20 | 100% | 0 | — | 159K | max_iterations |
| 02-clean-replay-library-25 | V2 | 25 | 100% | 0 | — | 223K | max_iterations |
| 03-classify-pipeline-25cyc | V3 | 25 | 100% | 0 | — | 214K | max_iterations |
| 04-tier1-smoke-6cyc | V4 | 6 | 0% | 0 | — | 59K | max_iterations |
| 05-autonomy-benchmark-21cy | V4 | 21 | 42% | 1 | 0.6022 | 267K | converged |
| 06-clean-run-14cyc | V4 | 14 | 64% | 1 | 0.6031 | 143K | converged |
| agent_derive | V5 | 23 | 61% | 1 | 0.6022 | 207K | converged |
| final | V6 | 13 | 92% | 2 | 0.6040 | 105K | converged |
| run07 | V4 | 20 | 50% | 1 | 0.6028 | 228K | converged |
| v5 | V5 | 30 | 45% | 0 | — | 303K | max_iterations |
| v5_exploit | V5 | 7 | 83% | 1 | 0.6014 | — | — |
| v5_exploit2 | V5 | 8 | 57% | 0 | — | 82K | max_iterations |
| v6 | V6 | 16 | 60% | 1 | 0.6031 | 166K | converged |
| v6b | V6 | 21 | 25% | 0 | — | 307K | token_budget |
| v6c | V6 | 22 | 88% | 1 | 0.6022 | 247K | converged |
| v6c_gpt55 | V6 | 13 | 62% | 1 | 0.6030 | 146K | converged |
| v6c_verify | V6 | 25 | 81% | 0 | — | 313K | max_iterations |
| v7 | V7 | 23 | 85% | 0 | — | 307K | token_budget |
| v7b | V7 | 25 | 86% | 0 | — | 286K | max_iterations |
| v8 | V8 | 5 | 100% | 0 | — | — | — |
| v8b | V8 | 25 | 40% | 0 | — | 466K | max_iterations |