03 / Pillars of Creation · Evaluation
A foundation should earn its handover.
A training run is not the finish line. We want a national team to receive a foundation with a clear account of what it can do, how it was built, and where it still fails.
Research agenda
Proposed work. Experimental results are forthcoming.
First, define the capability
Choose a bounded physical problem and a measurable success condition before training. Document the observation space, task distribution, and evaluation protocol so the result can be interpreted rather than merely advertised.
Test outside the training distribution
Our planned evaluations reserve unfamiliar objects, layouts, viewpoints, and task combinations. We will separate appearance changes from changes in geometry and physical interaction, and record which conditions expose failures.
Account for the whole experiment
Report data composition, pre-training compute, adaptation data, multiple runs, uncertainty, and downstream resource demands. Comparing foundations requires accounting for the work that came before the benchmark score.
Make the handover reproducible
Our intended acceptance process includes loading the checkpoint in a customer-controlled environment, reproducing a documented evaluation, and checking the development instructions without an Orion-hosted model endpoint.
Use adaptation as a measurement tool
A limited, repeatable adaptation probe helps measure the quality of pre-training. This is evaluation of our deliverable. The national team remains responsible for developing and operating its specialized systems.
Can the receiving team reproduce the result—and build the next one itself?