How it works

Data

ProteinGym's deep mutational scanning (DMS) benchmark, release v1.3, downloaded at a pinned version and checked against sha256 checksums. MutaScope keeps single substitutions only: 696,311 measured variants in 217 assays. Every mutation's wild-type residue is checked against the target sequence, and a mismatch stops the pipeline instead of being dropped.

Structures are ProteinGym's AlphaFold2 models, which match the assay sequences. 213 of the 217 assays have a usable one; the rest have none, so structure-aware models skip them (ADR 0003). A higher DMS_score always means fitter, following ProteinGym.

Models

Zero-shot models never see an assay's measurements. ESM-2 (35M, 150M and 650M parameters) scores a substitution by masking the residue and comparing the log-probabilities of the mutant and the wild type (ADR 0012). SaProt 650M does the same with tokens that combine each amino acid with its local 3D shape from the structure (ADR 0015). Their ensemble averages the two after standardising each within the assay (ADR 0016).

Supervised models are trained per assay on that assay's measurements, under ProteinGym's supervised protocol. The ladder runs from one-hot ridge regression through ridge on rich features (zero-shot scores, language-model embeddings, structure features), LightGBM and a small neural network (ADR 0019, ADR 0021). Each prediction comes with 80% and 90% intervals from CV+, a cross-conformal method (ADR 0024).

Evaluation

The main metric is Spearman correlation within each assay. Assays are aggregated as ProteinGym does (per protein, then per function type, then overall), never as a plain mean, with 95% intervals from a bootstrap over assays. Two models are compared only by a paired bootstrap of their difference on the same assays (ADR 0011, ADR 0014).

Before any modelling, the harness had to reproduce ProteinGym's published ESM-2 650M result from its own scores, and our ESM-2 650M scores had to agree with theirs in every assay (ADR 0010). Supervised models are scored on ProteinGym's random, modulo and contiguous folds; every search for settings stays inside the training folds, and each supervised model is compared with zero-shot on exactly the same test folds.

The results are on the Benchmarks page.

What the website shows, and why

Each assay shows one default model. 198 assays show the supervised model, Ridge, sequence + structure: they have a structure and at least 300 measured variants. The other 19 show a zero-shot model: SaProt 650M for the 15 smaller assays with a structure, ESM-2 650M for the 4 without one.

Why this rule, which was fixed before the final results (ADR 0025, ADR 0026):

• Choose on the task the website does. A full landscape is mostly predictions at positions with no measurement, which the modulo and contiguous splits test. There, ridge was as good as or better than LightGBM and the neural network, which win only when the position was seen in training.
• Supervise only where it pays. Below 300 variants no supervised model was shown to beat SaProt on unseen positions; from 300 up, all did (see the by-size chart on the Benchmarks page).
• Same model everywhere. Nothing is chosen per assay from its own results. Every prediction is labelled “Supervised (n = …)” or “Zero-shot”.

The live scoring page runs ESM-2 (35M by default, 150M on request) on ONNX Runtime on a small CPU server, so it is zero-shot only; its fast mode uses the one-pass wild-type marginal (ADR 0029, ADR 0031, ADR 0034).

Limitations

• Predictions are functional effects in laboratory DMS assays, each on its own scale. They are not pathogenicity predictions.
• Single substitutions only: no insertions, deletions or combinations of mutations.
• Supervised gains are three to six times smaller at unseen positions than at seen ones, and for Binding assays a gain on unseen positions is not shown at all (ADR 0026).
• Intervals have a constant width per assay: they say how wide the error usually is, not which variants are harder (ADR 0024).
• Two deployed targets lie mostly outside their structure, so most of their rows use imputed structure features; those rows are flagged (ADR 0026).
• Structures are AlphaFold2 predictions. Residues with low confidence (pLDDT below 70) carry no structure token for SaProt.

Two errors found in ProteinGym's SaProt scores

Our SaProt 650M scores had to match ProteinGym's published ones in every assay. In two assays they could not, and the cause was upstream (ADR 0015):

• POLG_HCVJF_Qi_2014: scored with a structure range shifted by one residue from an older reference file, so each residue carried its neighbour's structure token.
• BRCA2_HUMAN_Erwood_2022_HEK293T: scored in three structure chunks whose scores were written back in file order, which is not position order, so every row holds another variant's score.

For each, a test replays the wrong inputs through our scorer and reproduces ProteinGym's numbers exactly, which shows the difference is theirs. Both assays are left out of the agreement check only, and the benchmark uses our scores for them.

Read more

• Architecture decision records: every decision, with its evidence
• Model card (Hugging Face) and its source
• Dataset (Hugging Face) and its data card
• API documentation (OpenAPI; the server may take a moment to wake)
• Source code on GitHub