Benchmarks

Zero-shot models

Protein language models scored without any assay data: ESM-2 at three sizes, its wild-type marginal (one pass, what live scoring uses), SaProt (sequence plus AF2 structure tokens) and their ensemble. SaProt and the ensemble need a structure, so their comparisons run on the assays that have one.

Supervised models against zero-shot

Per-assay models trained on the assay's own measurements, under ProteinGym's five-fold cross-validation. Random folds put variants of the same position in training and test; modulo and contiguous hold out whole positions, so the model predicts residues it never saw. Each run is compared with a zero-shot model run through the same folds: SaProt for runs that use structure features, ESM-2 650M otherwise. The deployed model is Ridge, sequence + structure.

Gain per function type

Do the prediction intervals hold?

Each supervised prediction carries 80% and 90% intervals (CV+, a conformal method). Coverage is the share of held-out measurements that fall inside their interval, measured on test folds the intervals never saw. The calibration folds were either split-matched (split like the test: whole positions held out under modulo and contiguous) or random. The design was fixed before any coverage was computed (ADR 0024). The deployed intervals are split-matched.

The deployed landscapes also report coverage of measured variants by their out-of-fold intervals (0.800 at 80%, 0.900 at 90% over 198 assays). That holds by construction and is not evidence; the test-fold coverage above is (ADR 0026, weak point 11).