Module 7

Testing the engine

A numerical engine has a special testing problem: it can be wrong while looking completely healthy. No exception, no crash — just a beta of 2.3912 where 2.3883 was correct, flowing silently into a risk report. The beta service answers with four layers of confidence, each one a different kind of proof:

A known-value fixture — small, hand-checkable, frozen forever.
Reference cross-checks — the fast engine vs a slow textbook implementation, everywhere.
A performance proof — the complexity claim written as a failing test.
A concurrency smoke test — the thread-safety claims counted, not hoped for.

🎛 Fixture runner

Fixture data — 8 trading days (weekend absent)

date	03	04	05	06	09	10	11	12
MSFT	100	102	101	103	104	103	105	106
SPY	400	404	402	406	408	407	410	412

Corrupt one MSFT close

Corruption+0 → 105

This is the same fixture frozen in the Java service's RollingOlsEstimatorTest: 8 days of MSFT/SPY closes with expected betas {2.3883, 2.8729, 2.8728} computed independently in the design doc. The runner recomputes rolling beta from scratch — log returns, mean-centred cov/var over the trailing 3 returns, rounded half-up to 4dp. Because the expected values are frozen, a wrong number anywhere in the pipeline — a bad price, an off-by-one window, a rounding change — fails the test.

Layer 1 — the known-value fixture

The anchor of the whole suite is an 8-day MSFT/SPY dataset whose expected betas were computed independently, by hand, in the design doc — then frozen into the test:

private static final double[] MSFT = {100, 102, 101, 103, 104, 103, 105, 106};
private static final double[] SPY = {400, 404, 402, 406, 408, 407, 410, 412};

@Test
void knownValueFixtureFromDesignDoc() {
    // calcBeta(MSFT, SPY, 2021-08-10, 2021-08-12, 4): output indices 5..7, w=4.
    double[] betas = estimator.calc(pair(MSFT, SPY), 5, 7, 4);

    assertArrayEquals(new double[]{2.3883, 2.8729, 2.8728}, round4(betas));
}

From core-api/src/test/java/com/schonfeld/betaservice/calc/RollingOlsEstimatorTest.java.

The point of a known-value fixture is that the expected numbers did not come from the code under test. A test that asserts the engine equals the engine proves nothing; a test that asserts the engine equals three numbers worked out independently pins the entire pipeline — returns, alignment, windowing, rounding — to external truth. The widget above runs this exact fixture in your browser.

Layer 2 — the slow textbook reference, everywhere

Three hand-checked numbers cover three cases. The second layer covers all of them: the test suite carries a deliberately slow, obviously correct implementation — an explicit mean-centred cov/var loop — and compares the fast engine against it for every day at several window sizes, to twelve decimal places:

for (int w : new int[]{2, 3, 4, 5, 8, 20}) {
    double[] betas = estimator.calc(pair, 0, 7, w);
    for (int day = 0; day <= 7; day++) {
        double expected = referenceBeta(MSFT, SPY, day, w);
        ...
        assertEquals(expected, actual, 1e-12, "day " + day + " w " + w);
    }
}

From RollingOlsEstimatorTest.java (NaN branch elided).

Around this sit regression tests for the numerical edge cases Module 4 covered — including the nastiest one: a constant-but-nonzero baseline return, where the variance is mathematically zero but the computed denominator is cancellation noise of either sign. The test constantNonZeroBaselineReturnYieldsNaN generates 2,000 days of it and asserts the engine serves NaN — not the arbitrary huge ratio the noise would produce — on both the fast prefix-sum path and the small-window direct path. Bugs found once become tests forever.

Layer 3 — the performance proof

The engine claims warm queries are O(output days) — independent of the window size— thanks to its prefix-sum design. Claims in comments rot; claims in assertions don't. The performance test times the same 1,000-day query at w=20 and w=12,600 over ~80 years of synthetic history:

long smallest = bestOfFiveNanos(() -> service.calcBeta("TICK", "BASE", start, end, 20));
long largest = bestOfFiveNanos(() -> service.calcBeta("TICK", "BASE", start, end, 12_600));

// Naive O(n*w) would make this ratio ~600. Allow generous noise headroom.
double ratio = (double) largest / Math.max(smallest, 1);
assertTrue(ratio < 8.0,
        "w=12600 took " + largest + "ns vs w=20 " + smallest + "ns (ratio " + ratio + ")");

From core-api/src/test/java/com/schonfeld/betaservice/api/BetaServicePerformanceTest.java.

A naive O(n·w) loop would blow the ratio out to roughly 600×; the bound of 8× is generous enough to survive noisy CI machines yet far below the naive blow-up. If a future refactor quietly reintroduces a per-window inner loop, the build goes red — the complexity guarantee has become executable.

Layer 4 — the concurrency smoke test

Module 5 argued that the cache's coarse lock makes cold builds single-flight. The smoke test counts it: 16 threads × 50 iterations each hammer 6 pairs on a cold service, released simultaneously by a CyclicBarrier to maximize contention:

stampede.await(); // maximize contention on the cold cache
for (int i = 0; i < ITERATIONS; i++) {
    String[] pair = pairs[(offset + i) % pairs.length];
    double[] betas = service.calcBeta(pair[0], pair[1], start, end, 20);
    // exact bit equality (NaN-aware) with the serial run
    assertArrayEquals(expected.get(pair[0] + pair[1]), betas, ...);
}
...
assertEquals(pairs.length, stats.misses(), "each pair built exactly once (single-flight)");
assertEquals((long) THREADS * ITERATIONS - pairs.length, stats.hits());

From core-api/src/test/java/com/schonfeld/betaservice/api/ConcurrencySmokeTest.java (condensed).

Two things are asserted. First, every one of the 800 concurrent results is bitwise-equalto a single-threaded run — not “close”, identical. Second, the cache counters must read exactly 6 misses and 794 hits: each pair was built exactly once despite the stampede. If the lock were ever narrowed incorrectly, a duplicate build would show up as a seventh miss and the test would fail. Single-flight isn't claimed — it's counted.

Tests encode the contract

Step back and look at what the four layers pin down: the fixture pins the numbers, the reference pins the math everywhere, the flatness assertion pins the complexity, and the miss-count pins the concurrency behavior. Together they are the service's contract, written as executable checks. That is what buys the freedom this whole course has leaned on — swap the estimator, upgrade the cache to per-key futures, refactor the composition root — and if all four layers stay green, the behavior that matters is provably unchanged. Future refactors are free because the tests own the contract.

Things to try

• Run the fixture untouched — all three days PASS against the frozen expectations {2.3883, 2.8729, 2.8728}.
• Corrupt the 2021-08-11 MSFT close by just +1 and re-run — two of the three windows contain it, so two rows FAIL.
• Corrupt 2021-08-03 instead — everything still passes. The first close never enters the trailing windows of the output days, and the fixture knows it.
• Find the smallest corruption that still fails — even ±1 on one price moves the fourth decimal. That's how tight a 4dp fixture pins the pipeline.

← Module 6: Composition & DI Module 8: Python verification & load testing →