Module 5
Caching & concurrency
In the beta service, answering a query needs the ticker and baseline price histories aligned onto their common calendar. Building that aligned pair is O(total history) — a scan over decades of prices. But once built, every warm query on the same pair is cheap. That shape — expensive to build, cheap to reuse, bounded in number — is exactly the shape a cache is for. Not an unbounded map (memory grows forever), but a bounded LRU: keep the most recently used pairs, evict the least recently used when full.
The interesting engineering questions are the ones this module walks through: how little code an LRU really needs in Java, why the cache key deserves more thought than it usually gets, and what happens when two threads ask for the same cold pair at the same instant.
🎛 LRU cache simulator
Request a pair — get(ticker, baseline)
Cache contents
LRU (next to evict) → MRU
empty — every request will be a cold miss
Click a pair key to perform a cache get.
Hits
0
Misses
0
Evictions
0
Hit rate
—
size 0/3
This is exactly the Java service's PairCache: a LinkedHashMap(accessOrder=true) whose removeEldestEntry override evicts when size exceeds capacity. A hit moves the entry to the MRU end of the access order; a miss pays the expensive alignment build once and inserts the immutable result. The hit/miss/eviction counters mirror PairCache.stats(), the service's observability snapshot.
An LRU cache is ten lines of standard library
Java's LinkedHashMap has a rarely used third constructor argument: accessOrder=true. In that mode, every get moves the entry to the back of the internal linked list — so the front of the map is always the least recently used entry. Override one protected hook, removeEldestEntry, and the map evicts for you on insert:
this.cache = new LinkedHashMap<>(16, 0.75f, true) {
@Override
protected boolean removeEldestEntry(Map.Entry<PairKey, PairSeries> eldest) {
boolean evict = size() > maxEntries;
if (evict) {
evictions++;
}
return evict;
}
};From core-api/src/main/java/com/schonfeld/betaservice/store/PairCache.java (logging trimmed).
That's the whole eviction policy. No third-party cache library, no hand-rolled doubly linked list — the JDK collection already is an LRU when you ask it to be. The counters (hits, misses, evictions) are exposed through a stats() snapshot for observability — the simulator above mirrors it exactly.
The key is a value, not a string
The tempting shortcut is to key the cache by string concatenation — ticker + "|" + baseline. The service instead uses a two-field record, and the comment explains why:
/**
* Value-based key: record equals/hashCode compare the two fields
* directly, so no string encoding exists to collide (a concatenated
* "T|B" key would conflate {"A|B","C"} with {"A","B|C"}).
*/
private record PairKey(String ticker, String baseline) {
}From PairCache.java.
A record gives you correct equals/hashCode over the fields themselves — there is no flattened encoding that two different pairs could both map to. Note also that the key is the ordered pair: (A, B) and (B, A) are different entries, because direction matters — the baseline supplies the variance denominator of the beta.
One monitor, and single-flight falls out for free
Now the concurrency question. If two threads ask for a cold MSFT/SPYat the same instant, the naive check-then-build races: both see a miss, both pay the O(total history) build, and one result overwrites the other. The service's answer is deliberately coarse — one synchronized monitor covers the lookup and the build:
public synchronized PairSeries get(String ticker, String baseline) {
TickerSeries tickerSeries = store.series(ticker);
TickerSeries baselineSeries = store.series(baseline);
PairKey key = new PairKey(tickerSeries.ticker(), baselineSeries.ticker());
PairSeries cached = cache.get(key); // access-ordered: moves the entry to MRU
if (cached != null) {
hits++;
return cached;
}
misses++;
PairSeries pair = PairSeries.align(tickerSeries, baselineSeries);
cache.put(key, pair);
return pair;
}From PairCache.java (timing/log lines trimmed).
This makes same-pair requests single-flight by construction: the second thread cannot even re-check the cache until the first thread commits its build and releases the monitor — at which point the second thread's lookup is a hit. Each pair is built exactly once, ever, no matter how many threads stampede it. No double-checked locking, no atomics, no cleverness — the guarantee is a consequence of the lock's scope, and the concurrency smoke test in Module 7 proves it by counting misses.
The cost is stated, not hidden: while one thread builds a cold pair, threads wanting unrelatedpairs also wait. The class Javadoc calls this “a deliberate tradeoff” — adequate at this scale — and documents the production upgrade: per-key single-flight, caching a future per key installed atomically so only same-key waiters block (the Java Concurrency in Practice Memoizer, or Caffeine with weight-bounded eviction). Because the class has a single entry point, get, that upgrade is a drop-in behind the same signature.
The last piece is what makes the whole design safe: the cached PairSeries is immutable. Once a reference escapes the monitor, any number of threads can read it concurrently with no further locking — there is simply nothing to mutate. Immutability is doing the heavy lifting; the lock only has to protect the map itself.
Things to try
- • With capacity 3, request A/B, A/C, B/C, then A/B again — a hit, and it jumps to the MRU end.
- • Now request C/A — the cache is full, so the least recently used entry (A/C) is evicted, not the oldest-inserted.
- • Request A/B and then B/A — two misses. The ordered record key treats them as different pairs, because direction matters.
- • Shrink the capacity to 2 and keep cycling all five keys — watch the eviction counter climb. That's cache thrash: a working set bigger than the cache.