About these numbers
Provenance
The series on this page are re-scored through srbf.analysis from the archived raw
prediction snapshots of the Flash-ANSR v23.0 evaluation (the experiments
behind the paper Breaking the Simplification Bottleneck in Amortized Neural Symbolic
Regression), not yet from a fresh canonical srbf run. Flash-ANSR
series carry version v23.0 (or v23.3 for the KV-cache decode
re-evaluation); symbolic-data, srbf, and catalog artifact versions are -
because those packages did not exist at evaluation time. The version travels with each
series in the legend. The collapsible Ablations group exposes the paper's ablation
grid: training-data pipeline and budget (Track A), architecture reverts toward NeSymReS
(Track B), the constants prior (C1), and inference-only re-evaluations of the unchanged
checkpoint (Track Z). One ablation (B1, post-norm) is absent because its archived snapshots
carry no timing information and therefore cannot be placed on the wall-clock compute axis.
Who chose each configuration
Every series carries a configuration-provenance label, part of the benchmark's written
fairness policy. The labels answer the first question to ask of any
benchmark whose authors also enter their own method: who picked the hyperparameters?
Headline comparisons state the label of every entrant, and the labels also appear on
the compute-curve hovers and as table footnotes.
- Upstream defaults (PySR, NeSymReS, E2E): the method's own released
defaults, untouched in either direction. Where a default is consequential on these
benchmarks, srbf measures the consequence and documents it next to the method's
results rather than patching it.
- Author-blessed (all Flash-ANSR series, including ablations and
inference variants): the configuration was chosen by the method's authors. The
benchmark and Flash-ANSR share authors, which is exactly what this label discloses;
any method's authors can submit a blessed configuration on the same terms via the
fairness policy.
- Maintainer-chosen (the Prior reference): benchmark-native series
with no upstream and no external author.
Benchmarks and sweeps
Two benchmarks are available in the menu. FastSRB is a curated suite of
textbook and physics expressions evaluated at multiple sampled problem instances per
expression. v23-val is a held-out validation set of expression skeletons
drawn from the training distribution (in-distribution generalization, decontaminated
against training data). Three sweeps populate the x-axis choices: inference
compute, support size (how many data points the method sees), and
noise level (relative noise added to the support targets).
The compute axis is the median per-problem wall-clock fit
time: the one budget that is comparable across methods. Each method scales along
its own natural knob (Flash-ANSR sweeps the number of sampled candidates, PySR its
iterations, NeSymReS the beam width, E2E the candidate count), and plotting against time
unifies them into a single fair frontier. Times were measured on the same hardware per
sweep.
From raw predictions to a point on a curve
Every point on every curve is computed from per-problem raw predictions with the same
pipeline:
-
Placeholder rows are excluded. A benchmark draw that could not be
realized at all (e.g. no valid problem instance could be sampled for an expression at
the requested support size) is a placeholder and never enters any statistic.
-
Failures count against rate metrics, and only rate metrics. If a method
produced no usable prediction for a problem, the rate metrics (numeric recovery,
exact skeleton match, prediction success) score that problem as 0 rather than dropping it;
a recovery rate conditioned on success would be inflated and would vanish entirely where
success goes to zero (for example at a support size of 1). Diagnostic metrics
that are only defined when a prediction exists (FVU, token overlap, edit distances,
predicted lengths, timings) are computed over successful predictions only; the one
exception is the ground-truth skeleton length, a property of the benchmark itself,
which is recorded for every drawn problem.
-
Rows are grouped by ground-truth expression. Each benchmark expression
may be drawn several times (different sampled constants, support sets, or noise
realizations); the per-draw values are averaged into one value per expression, so no
expression is over-weighted.
-
The per-expression values are bootstrapped. The expressions are
resampled with replacement 1,000 times; the mean over expressions is computed for each
resample. The plotted point is the median of that bootstrap distribution and the shaded
band is its 2.5th–97.5th percentile range (a 95% confidence interval). The
n shown in a point's tooltip (tap or hover) is the number of
contributing expressions.
Reading the curves
Recovery and FVU are always scored against clean targets. In the noise
sweep the method fits on noisy support values, but its prediction is evaluated against the
noise-free function values: the question is whether the true function was recovered, not
whether the noise was reproduced.
Token-level and exact-match metrics compare raw prefix skeletons. The
archived predictions are not re-normalized to a common simplified form, and the baselines
tokenize variables differently, so cross-series comparisons of the symbolic-similarity
metrics (exact match, F1, edit distances) carry a systematic offset between method
families. Within a series, and across x for one series, they are consistent.
Some metrics exist only for Flash-ANSR series. The log-probability and
selection score come from the model's own decoder and candidate-selection stage; the
baselines have no equivalent, so their series are empty under those metrics.
Paired comparisons
Every method in this explorer solved the same benchmark expressions. The
Paired views (the Δ-curves, reached via Curves × Paired
Δ, and the verdict Matrix, via Table × Paired Δ) exploit that:
instead of comparing two methods' separate averages (whose uncertainty bands are inflated
by shared expression difficulty: hard expressions are hard for everyone), they
compute the difference per expression and ask whether those per-expression
differences are consistently one-sided. This also divides the work between the views: the
Absolute Table answers per-series questions (what does this method
achieve at this budget?), while A-vs-B questions get their calibrated answer in the
Paired views, because two marginal confidence intervals can overlap even when one
method is reliably better on expression after expression. If the question is "A vs B",
use a Paired view. The convention is
Δ = selected − baseline (in the Matrix, cells
read row − column), and the tap/hover record says which series a value favors. Every statistic on this page is computed ahead of time by
srbf's
open-source paired layer and published with the page: nothing is calculated in
your browser, so what you see is exactly what is citable. Every claim is per
benchmark: there is deliberately no combined number, because the two benchmarks measure different
populations.
How to read a verdict. Each comparison is judged against a
measurement-noise margin: the largest difference you would expect between two
equally good models, given these two series' own draw-to-draw noise. The margin
is derived from the data, never hand-picked: each model's repeated draws are split into
halves, the halves are compared against each other, and the margin is the 95th percentile
of those null differences, combined for the specific pair at hand. Verdicts are issued at
standardized compute budgets (≤1, 10, 100, or 1000 seconds per problem,
median; selectable in the view), and every method is evaluated at exactly
that time: interpolated per problem, linearly in log-time, between its two bracketing
measured configurations. This is the same model that draws the Δ(t) curves, so no
method is under- or over-credited just because its tested configurations happen to land
near or far from the budget. Two boundary rules: a method whose cheapest configuration
already exceeds the budget is n/a (it cannot run within t), and a method whose
measurements end below the budget is never extrapolated: its largest
tested configuration's value is carried forward, visibly flagged max tested,
and treated as a lower bound (more compute could only help it; the method is not
stagnating; we simply did not test beyond that point). A verdict stands only if
no max-tested side could overturn it by improving; otherwise it is downgraded to
undecided (max-tested limit). Same-method comparisons (ablations, the size
ladder of model scales) instead keep
the same measured configuration on both sides: comparing them at equal time
would let two factors vary at once. Every cell in the Matrix answers who is better,
at exactly t seconds?; every row of the Absolute Table answers what
does this method achieve, at exactly t seconds? The budget slider itself is
continuous for exploration: it snaps magnetically to the marked budgets, and any
position between them shows clearly-labeled descriptive reads of the
plotted curves; verdicts and corrected p-values belong to the pre-declared marks,
which keep the set of quotable claims finite and the corrections meaningful. Four
outcomes (all intervals are 95% confidence intervals):
- better / worse:
the confidence interval clears the noise margin: a real difference, larger than
benchmark noise can explain.
- equivalent: the interval fits entirely
inside the margin: any remaining difference is smaller than this benchmark can
measure. (This is measurement-equivalence, not proof the models are identical.)
- undecided: neither of the above, meaning not enough data to call it.
Each record reports its minimum detectable effect (MDE₈₀): the smallest
true difference this comparison would have caught reliably (80% of the time,
hence the subscript). Smaller differences could easily have been missed, so "undecided"
never means "equal".
Corrected vs raw p-values. Checking many comparisons at once produces
false positives by luck alone, so displayed significance is corrected by default:
Holm (strict) within the small pre-declared set of confirmatory comparisons,
and Benjamini–Hochberg (screening) for everything else, which stays labeled
exploratory. The confirmatory set: each ablation vs its parent model, adjacent
steps of the model-size ladder, and each model size vs each baseline (marked with a
small C in the matrix). The common thread is that every confirmatory
pair varies exactly one factor: a design choice, one scale step, or the method
itself. The p-value and the verdict answer different questions: the p-value
asks whether Δ is exactly zero, while the verdict asks whether the confidence
interval clears the pair's noise margin, so a comparison can be significant yet
undecided: a real-looking difference that is not provably larger than benchmark
noise. Verdicts are interval-based and do not change with the correction toggle, which
can show raw p-values for browsing, behind a notice; the corrected values are the
quotable ones. Every
cell records which family and method produced its correction, so numbers remain auditable
as the model roster grows.
Measurement details. The compute axis pairs configurations
whose median cost is t, not per-expression equal budgets; solid
dots are measured configurations, hollow dots are interpolated between neighboring
measured ones (never extrapolated). Where a model failed on an expression that its
opponent solved, rate metrics count the failure as 0 while diagnostic (quality-of-fit) metrics compare
only expressions both models solved; each record discloses those counts. On
v23-val, cross-method cells carry an in-distribution note: that benchmark is
drawn from flash-ansr's own training distribution. Bands on Δ-curves are
pointwise: each budget's band stands on its own, not as a statement about the whole curve. Displayed numbers are rounded to the precision their uncertainty justifies,
and every number on this page carries a release id; cite them together.
The Ranks view
The Ranks display answers the standings question: how do all
methods stack up at once? Within each benchmark expression, the fixed roster (every
Flash-ANSR size plus every baseline) is ranked 1st, 2nd, … like race placings, at
exactly the selected compute budget. Ranking within each expression makes hard
and easy expressions count equally (every expression hands out the same placings),
so a method's mean rank is a difficulty-adjusted standing. Ranks
measure consistency: winning by a hair counts exactly like winning by a mile,
which is why this view complements the Paired views (which measure by how much)
rather than repeating them.
Two statistical guards keep the picture honest. The Friedman test first asks
whether the spread of mean ranks is bigger than chance would produce between
interchangeable methods; if not, no groupings are drawn at all. If it is, the
critical difference (CD) is the smallest mean-rank gap that is meaningful after
correcting for comparing every pair at once; methods closer than one CD share a shaded
band, meaning no reliable rank difference was detectable at this sample
size: absence of evidence, not proof of equality (a pre-declared Paired
verdict can legitimately separate two methods that share a band: it asks a different,
magnitude-based question). Hollow dots mark methods whose measurements end below the
budget: their plotted position is a worst case that more compute could only improve.
Rankings exist for the continuous metrics that every roster method has; failures rank
last within their expression on quality metrics, while expression-property metrics
(like length) are ranked only over expressions every method solved, a smaller,
explicitly-labeled population. One metric is primary and quotable
(held-out log₁₀ FVU); every other metric's ranking is
exploratory browsing; the metric menu is grouped accordingly, and a small badge on the
Metric control says which kind is on screen.
Metrics
Definitions for every metric in the menu, following the evaluation protocol of the
Flash-ANSR
paper, extended with a few additional diagnostics. Notation: a benchmark problem consists of a ground-truth expression with
prefix-notation skeleton \(\bar{\tau}\) (numeric constants masked to a
<constant> token, structure normalized with SimpliPy), a
support set \((X, y)\) of \(N\) points given to the method for fitting, and a
disjoint, equally sized validation set \((X_{\mathrm{val}}, y_{\mathrm{val}})\)
from the same input domain that the method never sees. A method returns a predicted
expression with skeleton \(\hat{\tau}\) and fitted constants; \(\hat{y}\) and
\(\hat{y}_{\mathrm{val}}\) are its values on the support and validation inputs.
\(|\tau|\) denotes the length of a skeleton in prefix tokens. Aggregation across problems
works as described in About these numbers; the arrows note which
direction is better.
Numeric fit
All numeric metrics build on the Fraction of Variance Unexplained, the
scale-invariant normalized squared error
$$
\operatorname{FVU}(y, \hat{y}) \;=\;
\frac{\sum_{i=1}^{N} (y_i - \hat{y}_i)^2}{\sum_{i=1}^{N} (y_i - \bar{y})^2},
\qquad \bar{y} = \tfrac{1}{N} \textstyle\sum_{i=1}^{N} y_i .
$$
An FVU of 0 is a perfect fit, an FVU of 1 is no better than predicting the mean, and
values above 1 are worse than the mean predictor. Because the denominator is the variance
of the targets, the measure is invariant to the output scale of the problem. The
implementation guards the numerics: a divergent or non-finite prediction scores
\(\infty\), and residuals are rescaled internally to avoid floating-point over- and
underflow.
Numeric recovery (vNRR) higher is better
$$
\mathrm{vNRR} \;=\; \frac{1}{K} \sum_{k=1}^{K}
\mathbb{1}\!\left[ \operatorname{FVU}\!\big(y_{\mathrm{val}}^{(k)},
\hat{y}_{\mathrm{val}}^{(k)}\big) \le \varepsilon_{32} \right],
\qquad \varepsilon_{32} = 2^{-23} \approx 1.19 \times 10^{-7}
$$
The validation Numeric Recovery Rate, the headline metric. A problem counts as
recovered only if the predicted expression matches the clean function values on the
unseen validation points up to 32-bit floating-point machine precision. This is
a deliberately strict, essentially all-or-nothing criterion: an expression either is the
(numeric) ground truth on the domain or it is not. Approximate fits, however close, do
not count. Failed predictions count as 0.
Numeric recovery on support (fNRR) higher is better
The same machine-precision indicator evaluated on the support points the method was
fitted on, \(\mathbb{1}[\operatorname{FVU}(y, \hat{y}) \le \varepsilon_{32}]\).
Comparing fNRR to vNRR separates memorization of the support from generalization to the
rest of the domain: an expression that nails the support but misses validation is an
overfit, not a recovery.
log10 FVU (validation / support) lower is better
$$
\log_{10} \operatorname{FVU}\big(y_{\mathrm{val}}, \hat{y}_{\mathrm{val}}\big)
\quad\text{and}\quad
\log_{10} \operatorname{FVU}\big(y, \hat{y}\big)
$$
The continuous companion to the recovery rate: how close the fits are on a log scale,
including approximate fits the strict \(\varepsilon_{32}\) threshold discards. A value
of \(-2\) means 99% of the target variance is explained; \(-7\) is at the recovery
threshold. Both are diagnostics over successful predictions only, and cover fits with
finite, positive FVU: an exact \(\operatorname{FVU} = 0\) maps to \(-\infty\)
and drops that expression from this aggregate, and a divergent fit
(\(\operatorname{FVU} = \infty\)) likewise drops out. Note that the paper goes further
and excludes all numerically perfect fits
(\(\operatorname{FVU} \le \varepsilon_{32}\)) from its log-FVU summaries; this page
keeps near-perfect fits with tiny positive FVU, so its levels are not directly
comparable to the paper's.
Success
Prediction success rate higher is better
The fraction of problems for which the method produced any usable prediction at all:
decoding, parsing, compiling, and constant fitting all completed and returned an
evaluable expression. This is the denominator-integrity metric behind the failure
convention above: where success dips, the rate metrics absorb the failures as zeros and
the diagnostic metrics thin out (visible as shrinking n in the tooltip).
Symbolic similarity
These metrics compare the predicted skeleton \(\hat{\tau}\) to the ground-truth skeleton
\(\bar{\tau}\) as token sequences and trees. They are computed on raw prefix
skeletons (see the caveat in About these numbers): the predictions
are not re-normalized to a common simplified form, so cross-series levels are offset and
the interesting signal is the trend within a series.
Exact skeleton match (raw) higher is better
$$
\mathrm{SRR}_{\mathrm{raw}} \;=\; \frac{1}{K} \sum_{k=1}^{K}
\mathbb{1}\!\left[ \hat{\tau}^{(k)} = \bar{\tau}^{(k)} \right]
$$
Token-for-token equality of the two prefix skeletons. This is the most conservative
symbolic criterion on this page: a prediction that is mathematically identical but
written differently (e.g. re-ordered commutative operands, an unsimplified sub-term)
does not count. It is a raw-form variant of the paper's Symbolic Recovery Rate, which
re-simplifies both sides before comparing. Failed predictions count as 0.
Skeleton token F1 / precision / recall higher is better
$$
P = \frac{|\mathcal{T}(\hat{\tau}) \cap \mathcal{T}(\bar{\tau})|}{|\mathcal{T}(\hat{\tau})|},
\qquad
R = \frac{|\mathcal{T}(\hat{\tau}) \cap \mathcal{T}(\bar{\tau})|}{|\mathcal{T}(\bar{\tau})|},
\qquad
F_1 = \frac{2PR}{P + R}
$$
Overlap of the sets of unique tokens \(\mathcal{T}(\cdot)\) used by the two
skeletons: did the prediction use the right operators, variables, and constants at all?
Precision penalizes inventing tokens the ground truth does not use; recall penalizes
missing tokens it does use. Being set-based, these ignore token order and multiplicity:
they measure vocabulary agreement, not structure. \(F_1\) is defined as 0 when both
\(P\) and \(R\) are 0.
Skeleton edit distance (normalized) lower is better
$$
d_{\mathrm{edit}}(\hat{\tau}, \bar{\tau}) \;=\;
\frac{\operatorname{Levenshtein}(\hat{\tau}, \bar{\tau})}{\max(|\hat{\tau}|, |\bar{\tau}|)}
\;\in\; [0, 1]
$$
The Levenshtein distance between the two prefix token sequences (minimum number of
token insertions, deletions, and substitutions to turn one into the other), normalized
by the longer length. Unlike the set-based F1, this is order- and count-sensitive: it
measures how much of the sequence structure is wrong, not just the vocabulary.
Tree edit distance (ZSS) lower is better
The Zhang–Shasha tree edit distance between the expression trees obtained by
parsing the two prefix skeletons: the minimum total cost of node insertions, deletions,
and relabelings that transforms one tree into the other. In this implementation the
operations are weighted by the character-level string distance of the node labels
(inserting or deleting a node costs the length of its label; relabeling costs the string
edit distance between the two labels), so long-named tokens like
arccosh or <constant> weigh more than single-character
ones, and the value is unnormalized label-edit cost, not a node count. Because it works
on the parsed tree rather than the flat token sequence, it respects the expression
structure: swapping a deeply nested operator costs the same as swapping a shallow one,
and subtree moves are scored more faithfully than by string edits.
Complexity and parsimony
Expression length ratio (pred/GT) 1 is ideal
$$
\rho \;=\; \frac{|\hat{\tau}|}{|\bar{\tau}|}
$$
The length of the predicted skeleton relative to the ground truth, in prefix tokens.
A ratio of 1 means the prediction is exactly as concise as the true expression; values
above 1 flag verbose, over-parameterized predictions (a common failure mode: fitting
the data with a longer expression than necessary), values below 1 flag predictions that
are too simple to be the true law. This is the second headline metric because it tracks
interpretability, the point of symbolic regression: a recovered-looking fit
made of twice as many terms is a worse scientific answer.
Length ratio |log2| error lower is better
$$
\left| \log_2 \rho \right| \;=\; \left| \log_2 \tfrac{|\hat{\tau}|}{|\bar{\tau}|} \right|
$$
The symmetric version of the length ratio: doubling and halving the true length are
equally wrong (both give 1), and 0 means exactly matched length. Useful when averaging,
where over- and under-shoots of the raw ratio would otherwise partially cancel.
Predicted / GT skeleton length lower is more concise
The raw prefix-token lengths \(|\hat{\tau}|\) and \(|\bar{\tau}|\) behind the ratio.
The ground-truth length is a property of the benchmark (constant per expression, useful
as a reference line); the predicted length shows how expression size scales with
compute, noise, or data availability.
Constant-count ratio / delta (pred vs GT) 1 / 0 is ideal
$$
\frac{c(\hat{\tau})}{c(\bar{\tau})}
\qquad\text{and}\qquad
c(\hat{\tau}) - c(\bar{\tau}),
$$
where \(c(\cdot)\) counts the <constant> tokens in a skeleton. A
positive delta means the prediction uses excess constants: free parameters
that let a structurally wrong expression bend toward the data. The ratio is only
defined for ground truths that contain at least one constant; the delta is defined
everywhere.
Unary-nestedness delta (pred − GT) lower is better
Total nestedness counts, over each maximal chain of \(m\) directly nested unary
operators, the excess \(m - 1\), summed over the expression:
\(\sin(\log(x))\) contributes 1, \(\sin(\log(\operatorname{arccosh}(x)))\) contributes
2, and \(\sin(x) + \log(x)\) contributes 0. Deeply nested unary chains
(\(\exp(\sin(\exp(\dots)))\)) are a hallmark of unnatural, hard-to-interpret
predictions. The reported value is the difference between the predicted and
ground-truth nestedness, so 0 means the prediction is as (un)nested as the true
expression; this is a delta variant of the paper's Total Nestedness, which reports the
predicted expression's absolute count.
Model internals (Flash-ANSR series only)
Predicted log-probability higher is better
The log-probability \(\log p(\hat{\tau} \mid X, y)\) of the selected candidate's token
sequence under the model's decoder, summed over the tokens of the sequence: how likely
the model itself considered the expression it ended up returning. Being a sum, its level
scales with expression length. Baselines have no decoder likelihood, so their series are
empty here.
Predicted selection score lower is better
$$
s(\hat{\tau}) \;=\; \log_{10} \operatorname{FVU}(y, \hat{y})
\;+\; \gamma \, |\hat{\tau}|,
\qquad \gamma = 0.05
$$
The parsimony-penalized criterion Flash-ANSR uses to pick its final answer from the
sampled candidate set: the best candidate is the one minimizing support-set log-FVU
plus a complexity penalty of \(\gamma\) per token (each additional token must be
bought with roughly an 11% relative FVU reduction). The value plotted is the score of
the selected candidate; it is a model-internal quantity, shown for completeness and
for reasoning about the selection stage.
Cost
Per-problem fit time (s) lower is better
End-to-end wall-clock seconds to solve one problem: for Flash-ANSR that spans encoding
the data, decoding candidates, constant fitting, and candidate selection; for the
baselines, their full search or inference pipeline. The median fit time of a run is
also what defines the compute axis, so this metric makes the x-axis itself inspectable
against noise level or support size.
