Evaluation instrument / 01

Paired evaluation chamber

Compare code-generating systems task by task. Estimate repeated-sampling success, preserve paired outcomes, and inspect uncertainty before declaring a winner.

Unbiased pass@k
Paired bootstrap
Deterministic seed
Bounded execution

Live calculation

One claim. Two systems. Visible uncertainty.

No model calls. No uploaded code. The API evaluates bounded Boolean outcome vectors only.

Repeated generation Total samples Correct samples k attempts

Paired task outcomes

Enter equal-length binary sequences. Each position is the same task for both systems; 1 means pass and 0 means fail.

System A outcomes System B outcomes

Confidence Bootstrap resamples Seed

pass@k

— Probability of at least one correct sample

Accuracy delta

— System A minus System B

Confidence interval

— Paired percentile bootstrap

A ahead in replicates

— Empirical bootstrap frequency, not a posterior

Method card

What this evidence can—and cannot—support

Repeated sampling

pass@k uses the unbiased without-replacement estimator popularized by code-generation evaluations. It is not the single-sample pass rate.

Paired comparison

Both systems are resampled on the same task indices. This preserves task difficulty instead of pretending the two result sets are independent.

Interval, not theater

The percentile interval exposes uncertainty in the observed task set. It does not establish generalization to a different benchmark or deployment distribution.

Reproducible boundary

The seed, confidence level, resample count, and bounded input vectors are explicit. Identical requests produce identical receipts.