Distributed Cache

Caching is a powerful and common technique that reduces load on downstream databases as well as latency. In a microservices environment, where there are many replicas of the same microservices, it is often needed to share the cache among replicas. One replica might store an item in the cache, while another replica might load it. An invalidation of a cached element by one replica needs to be visible to all others.

Localized Cache

In Microbus, each microservice holds in-memory an LRU cache that is shared with all peer replicas of the microservice, but not with other microservices. Each replica’s local LRU cache is a segment of the entire cache, and every key belongs to exactly one segment: the cache assigns each key an owner replica, and a Store or Load is a single addressed message to that owner rather than a broadcast to every peer.

The capacity of the cache scales horizontally with the number of replicas of the microservice, and because each key lives on one replica only, that scaling costs no cluster-wide fan-out per operation.

The cache is scoped to a single microservice, therefore isolating it from side-effects that can be caused by “noisy neighbor” microservices. Isolation also makes it possible to independently scale to the individual needs of each microservice.

Key Ownership

A key’s owner is computed, not assigned. Each replica derives it locally as the peer that scores highest for that key under rendezvous hashing, which is a deterministic function of the current peer set. Two replicas that agree on who is present therefore agree on where every key lives, with no coordinator, no election, and no version service — only the peer set has to be agreed on; the key-to-owner map never does.

Replicas learn the peer set by broadcasting a discovery ping, periodically and immediately whenever a request to an owner times out. From the sorted set they derive a generation, so any membership change is observable as a generation change.

During a change two replicas briefly disagree, and the cache tolerates that rather than preventing it. Every request carries the caller’s generation, and the owner accepts it if it matches its own current generation or, within a short overlap window, its immediately previous one. A Load that misses during the window retries once against the previous generation’s owner. When ownership moves, the old owner ships the displaced keys to the new one and drops its own copy.

Every failure this machinery can produce is a miss, never a stale read: a rejected write is dropped and recomputed, a stranded key is recomputed. The one exception is a network partition, where each side derives its own membership and the same key can be written independently on both. Preventing that would require exactly the coordination this design avoids, and it is unavoidable for a cache that stays available under partition — which is why “cache only what you can afford to lose” is a hard rule here rather than a suggestion.

Stampede Protection

When many concurrent requests miss the same cache key at once - a “thundering herd” or cache stampede - a naive cache lets every requester race to recompute the same value, hammering the underlying data source. The cache’s LoadOrCompute and GetOrCompute operations protect against this using singleflight: concurrent callers in the same process for the same key share a single maker invocation; only one goroutine computes the value while the rest wait for the result. Stampede protection is per-process, so with N replicas up to N concurrent maker invocations may still occur on a cold key - but the load on the data source is bounded to roughly one regeneration per key per replica rather than one per request. Maker errors are not cached, so a transient failure does not poison the cache for the rest of the TTL.

Data can survive a clean shutdown of a microservice if there is at least one other replica running at that time that has enough capacity to hold its data.

Cached elements can get evicted for various reason and without warning. Cache only that which you can afford to lose and reconstruct from the original data source. A distributed cache is not shared memory. Do not use a distributed cache to share state among peers.

The Trouble with a Centralized Cache

Using a centralized cache is a common anti-pattern that may result in system instability or even an outage.

A centralized cache shared by multiple microservices creates a dependency among those seemingly unrelated microservices. For example, a misbehaving microservice can overwhelm the cache, resulting in evictions of elements cached by other microservices. Those in turn will experience excessive cache misses and will have to hit their data stores again and again. This can easily bring down the system to its knees or worse.

Similarly, a centralized cache is a bottleneck and a single point of failure (SPOF). If it is overwhelmed, fails or restarted, all microservices using that cache will be affected at the same time. This too will result in a high number of cache misses and consequently a high load on the data stores.

It is also a matter of security when multiple microservices can read and write to the same cache. For example, a compromised microservice may be able to access user access tokens stored in a centralized cache by the authentication microservice.

A centralized cache often does not allow for setting a different TTL or memory limits on a per-microservice basis. The “SLA” is the same for all clients.

A cache that is localized to a single microservice is isolated from other microservices. The blast radius of a failure is limited to that microservice alone.