Brian Hung
Projects
Evals
A TypeScript framework for evaluating language models, built on the Vercel AI SDK.

A · the platform

Model calls arrive from a provider proxy, the AI SDK, OpenTelemetry, or any service that can post JSON. They land in one store you host and are graded as they arrive. From there a call can be judged, replayed as a new example, gated in CI, or exported.

online scorers rate each call as it landsProvider proxyswap a base URLAI SDK appstraceAISDKModelOpenTelemetryOTLP to /v1/tracesOther proxiesPOST /api/captureevals serveone store you host: files, SQLite or D1viewer · review · feedback · API · MCPScorejudges over stored callsReplaycalls become eval dataRun and gateCLI, vitest, CIExportOTLP, webhooks, HTMLRuns execute locally, on Temporal, or on Cloudflare Workflows, and write back to the same store.

B · the loop, as Visual Lookup runs it

Visual Lookup's prompts were tuned this way. GEPA proposes a prompt, the framework scores it on 222 photos, and a reflection step reads the misses and writes the next prompt. Each candidate is one run, so any of them can be opened, compared and traced.

prompta record per photomissesa better promptcandidate promptGEPA proposes oneevals runone run per candidatescorescity and name, half eachreflectionreads why each miss missedgolden set222 photosPlaces responses are cached, so scoring another candidate costs only model tokens.

C · dataset, runner, actor

Every run has the same core. The runner hands examples to the actor a few at a time; the actor calls the model and returns metrics for each. Every result is saved as it lands, so a stopped run picks up where it left off.

Runnera few examples at a time; a rerun skips what is doneexampleappendedat the endDatasetid, input, expectedactor.act(example)returns metrics, metadataSamplerthe model, every call tracedresponsesstored as JSONLtraceswhat the model sawactor.aggregaterun metricsReportersconsole, webhook, CI gate

D · with a sandbox and an external harness

Not built yet. A coding agent such as Claude Code or Codex runs its own tool loop, so the actor stops driving the model and starts managing a workspace. Each example gets a fresh sandbox with its repo restored. The agent works in it; graders run the tests there before the sandbox is destroyed.

Runnera few examples at a timeSandboxa fresh one per exampleat the endDatasettask, repo snapshotrestore the repofrom its snapshotHarnessAgent.generateClaude Code or Codexgraderstests and checks, in the sandboxtracesreportGenerateResultresponsestests passed, the diffactor.aggregaterun metricsReportersconsole, webhook, CI gateThe harness runs its own tool loop in the sandbox. The actor sees its stepswhen the turn ends, and grades the workspace before the sandbox is destroyed.