Floating point & the noise floor

The real guard: a relative noise floor, not == 0

Checking denominator == 0 is useless — cancellation noise is almost never exactly zero. The engine instead asks whether the denominator is meaningfully large relative to the terms it was cancelled from:

private static final double VARIANCE_NOISE_FLOOR = 1e-9;

double denominator = m.count() * m.sumYY() - m.sumY() * m.sumY();
double noiseFloor = VARIANCE_NOISE_FLOOR
        * Math.max(m.count() * m.sumYY(), m.sumY() * m.sumY());
if (Double.isNaN(denominator) || denominator <= noiseFloor) {
    return Double.NaN; // baseline variance is zero/noise (or data was NaN)
}
return (m.count() * m.sumXY() - m.sumX() * m.sumY()) / denominator;

From RollingOlsEstimator.betaAt() — core-api, com.schonfeld.betaservice.calc.RollingOlsEstimator

The floor is relative: 1e-9 times the cancelled terms, not an absolute cutoff. That's about 1,000× the ~1e-12 relative error prefix differencing leaves on decades-long series, while real return variance sits many orders of magnitude above it — so the check swallows noise without ever eating a legitimate beta. At or below the floor, the honest answer is NaN.

Tiny windows get a different treatment entirely

A 5-day window divides by a near-zero variance, so the ~1e-12 prefix error stops being invisible — on 40-year series it was enough to flip the 4th decimal of the served beta. Rather than tolerate that, small windows skip the prefix trick and compute directly, mean-centred:

/** Windows with at most this many returns take the direct path. */
private static final int SMALL_WINDOW_RETURNS = 16;

if (n <= SMALL_WINDOW_RETURNS) {
    return smallWindowBeta(pair, firstReturn, lastReturn);
}

RollingOlsEstimator — the small-window branch (still bounded O(1) per output day)

The direct path reads each return via an adjacent prefix difference (error ~1 ulp per element instead of ~1e-12 on the whole window sum) and centres on the mean before multiplying — numerically exact where it matters, and a bounded constant cost since n ≤ 16.

And the last rounding is done properly

The service publishes betas to 4 decimals. Binary floats can't represent most decimal fractions, so decimal rounding goes through BigDecimal with an explicit, symmetric rule:

public static double round4(double value) {
    if (Double.isNaN(value) || Double.isInfinite(value)) {
        return value;
    }
    return BigDecimal.valueOf(value)
            .setScale(4, RoundingMode.HALF_UP).doubleValue();
}

Rounding.round4() — core-api, com.schonfeld.betaservice.calc.Rounding

HALF_UP is symmetric — ties round away from zero, so −0.00005 becomes −0.0001 — and NaN/infinities pass through untouched because BigDecimal.valueOf would throw on them. Because none of these failure modes announce themselves, the service also cross-checks every result against an independent reference implementation: the only reliable way to catch a bug whose symptom is a plausible-looking number.