How to read the results

• Choosing a prediction.
• Judging it.
• Problems. Each problem is one known formula, with 512 data points given to the method and 512 held-out points used only to check the formula it returns. The points are sampled from the ranges the problem set specifies, without noise. Every method is run twice on each problem, with new points each time.
• Settings. Who chose each method's settings is shown next to its name in the explorer (what the labels mean).
• Versions.
• Time. Seconds per problem. (budgets and time)
• Statistics. How problems without a usable formula count, why some points are missing, what the 95 % intervals mean.

Each problem is one known formula, the ground truth, taken from one of 29 published problem sets, 6,660 problems in total. For each problem, 512 input points are sampled from the ranges its problem set specifies, and the formula's values there are computed without noise. These are the given points (the support set), the only data the method sees. 512 further points are sampled the same way and kept back: the held-out points (the validation set), used only to check the formula the method returns. The documentation lists every problem set with its source. The largest, erbench-syneq (machine-generated formulas), holds 5,301 of the 6,660 problems, so an average over all problems mostly reflects it; the Problem sets tab of the explorer shows each set on its own.

Every method is run twice on every problem, each time with newly sampled points. The curves and tables average over the runs that are finished: both, once the second is complete.

The neural methods here learn from machine-generated formulas before they see a problem. If a benchmark formula were among them, a method could recall it instead of finding it. The problem sets come from symbolic-data, a companion package that writes every formula in one standard form. A training-data generator built on it can therefore exclude every benchmark formula, also when it is written with other variable names or other constant values, and srbf can check afterwards, problem by problem, that the exclusion covers the benchmark (srbf decontamination). Removing test data from training data like this is called decontamination.

Flash-ANSR's training data is generated this way, with all 29 problem sets excluded. E2E and NeSymReS were trained by their authors on data of their own, which this benchmark cannot check. Like their training formulas, the formulas of erbench-syneq, the largest problem set, are machine-generated; the Problem sets tab of the explorer shows how the methods fare on each set.

Every method is allowed to return one formula per problem, its prediction. How it picks that formula is up to the method; in the explorer, the ? after a method's name describes its rule.

Every prediction is then checked in two ways. Numerically: does it reproduce the held-out points? Symbolically: is it the same formula as the ground truth? For the symbolic check, the variables of both formulas are named by their input column, and both are rewritten into a standard form by SimpliPy, a simplification library, and then compared symbol by symbol; depending on the metric, their numbers are ignored or not (see Metrics). The check is strict: two formulas that always give the same values, but that SimpliPy does not bring into the same form, count as different.

Most methods find better formulas when they may search longer. Each method is therefore run at a series of budgets, powers of two: 1, 2, 4, 8 and so on (the Flash-ANSR models skip 32,768 and end at 65,536). The budget is counted in the method's own unit: the number of candidate formulas it may generate (Flash-ANSR, E2E), its beam width (NeSymReS: how many partial formulas its search keeps), or its number of search iterations (PySR). Each method runs over its own range of budgets; the explorer says which budgets a method runs at wherever one is missing.

Because these units differ, the page also shows what each budget costs: the seconds per problem, measured for every method on the same workstation (16 CPU cores, one RTX 4090 GPU), one method at a time, on a fixed sample of 262 problems. Only these times are shown. Most results are computed on a shared compute cluster, where the time a job takes depends on the machine it lands on, so those times cannot be compared. The problem sets' average times are combined like every other value (see How values are averaged), and problems without a usable formula are not timed. A method that has not been timed yet can only be drawn against its budget. Candidate counts and beam widths share the budget axis of the curves; PySR's iterations do not fit on it, so PySR's curves are drawn against time. Tables and the same-budget ranking list PySR by its number of iterations.

The runs are still in progress, so results arrive over time, one problem set and budget at a time, and each method runs over its own range of budgets. A point, a table value or a rank is shown only once the method has results for every problem set you selected, at that budget. Then every value covers the same problems. A problem set counts as done once each of its problems has at least one finished run. Each problem is run twice, and its value is the average of its finished runs, so while the second runs finish, some problems rest on one run.

The reason is that problem sets differ a lot in difficulty. An average over whichever problem sets happen to be finished would mostly show which ones are done, not how good the method is. To see partial results, select fewer problem sets in the explorer: a point appears as soon as the selected ones are finished. The Progress page shows how far each method has come, in results and in measured times, budget by budget.

Sometimes a method returns no usable formula for a problem: it crashes, or returns something that cannot be evaluated. How such a problem counts depends on the metric. In a rate, such as Numeric Recovery, it counts as a miss. In the overlap metrics (Token Overlap and Variable Overlap), which range from 0 to 1, it counts as 0, the worst value; the explorer can leave such problems out instead.

All other metrics, such as the fit error or the formula's length, are measured only on the problems where the method returned a usable formula; most of them have no worst value, since a formula can always be worse. A method that fails on the hard problems then looks better than it is, because only its easier problems are measured. The explorer therefore draws a point hollow when it is based on fewer than 90 % of the problems; you can move this threshold.

The problems come in problem sets, and the sets differ a lot, in difficulty and in size: the largest holds 5,301 of the 6,660 problems. A plain average over all problems would mostly describe that one set. Every value is therefore averaged in three steps. First, each problem's runs are averaged, so every problem counts once. Second, each problem set's problems are averaged. Third, the problem sets are averaged, each weighted by how precisely its own average is known, plus an allowance \(\tau^2\) for how much problem sets differ from each other:

$$ \hat{\theta} = \frac{\sum_s w_s\, \bar{x}_s}{\sum_s w_s}, \qquad w_s = \frac{1}{\tau^2 + \sigma_s^2 / n_s}, $$

where \(\bar{x}_s\) is the average of problem set \(s\) over its \(n_s\) problems and \(\sigma_s^2\) the variance of its problems' values. The allowance \(\tau^2\) is estimated from the data (Paule and Mandel's estimator), the usual way to combine studies that measure the same quantity on different populations. Problem sets differ far more than chance would make them differ, so the allowance dominates: every set counts about the same whatever its size, and a set of only a few problems counts less, because its own average is uncertain.

Rates are averaged on the log-odds scale, which keeps them and their intervals between 0 and 100 %. Ratios, such as the MDL ratio, are averaged on a log scale, as geometric means: a ratio of 2 and one of 0.5 average to 1. Medians and distributions weigh each problem set as the average does.

Every value comes with a 95 % interval. It shows how precisely the value is known on the problem sets you selected: the weights of the sets are held fixed, and only the variation between the problems within each set counts. With \(v_s = \sigma_s^2 / n_s\), the squared standard error of set \(s\)'s average,

$$ \hat{\theta} \;\pm\; t_{0.975}\, \frac{\sqrt{\sum_s w_s^2\, v_s}}{\sum_s w_s} , $$

with \(t\) on \(\sum_s (n_s - 1)\) degrees of freedom, close to 1.96 here. The interval is about srbf's problem sets, not about problem sets in general. Tables and tooltips also show where the value of one more problem set would fall, with 95 % probability, once at least three problem sets are selected. That range is much wider than the interval: the interval says how precisely the average is known, the range how far single problem sets spread around it.

The people who run this benchmark also develop one of the methods, Flash-ANSR. The first question to ask of such a benchmark is: who chose each method's settings? Every method therefore carries a label that answers it, shown next to its name in the explorer. The labels are part of the written fairness policy.

• Upstream defaults (PySR, NeSymReS, E2E): the settings the method's own release ships with, not tuned in either direction. srbf sets only the budget and, for PySR, the operators: the 23 the problems are written in, where PySR's own default is +, −, × and ÷.
• Author-blessed (the Flash-ANSR models, the combination with PySR and the random-guessing reference): the settings were chosen by the method's authors. Here they are also the benchmark's authors, and this label says so. The authors of any other method can submit their own settings the same way.
• Maintainer-chosen: settings chosen by the benchmark's maintainers, for a method that has no release and no outside author of its own.