Brian Hung
Projects
Spreadsheet SWE-bench
A SWE-bench-style benchmark: implement one spreadsheet function in a Rust engine, graded by its tests.
Workdir
a fresh clone per prediction, deleted after
instance
one row of the dataset
prediction
instance_id, model_patch
path check
gold's files, never tests
git clone
IronCalc at base_commit
setup_patch
stubs out the function
model_patch
the prediction's diff
reset the tests
then apply test_patch
cargo test
FAIL_TO_PASS, PASS_TO_PASS
resolved
if every test passes
Most tasks start from a working engine; setup_patch takes the function out.