How a Redis Stampede (Thundering Herd) Hit Our OTA Search Ranking Engine — And How We Fixed It

The Problem

Our OTA platform lets admins set a global Airline Rank that decides how flight results are ordered in the search feed — independent of which of our 30 integrated GDS/NDC suppliers a given offer came from. A single search fans out to all 30 suppliers and comes back with 200-300 feeds total, but those feeds only span around 40-50 unique airlines. To sort the merged results by rank, the search pipeline looks up each of those 40-50 airlines' rank once per search, checking Redis first.

On a cache hit, that's fast — 40-50 quick Redis reads and the results are sorted. But on a miss, the lookup falls through to the database, and two things turned occasional misses into a full-blown Redis stampede, a classic thundering herd:

• Unranked airlines had nothing to cache. If admin had never set a rank for a given airline, there was no cached value to represent that — positive or negative. So every single time that airline showed up in a search's results, it was a guaranteed database hit, forever.
• Concurrent misses had no coordination. When many searches missed on the same airline at the same moment — a newly onboarded airline, a cache flush, a burst of traffic on a popular route — every one of those requests independently raced to the database for the exact same value, all at once. Instead of one query, the database got hit by a herd of identical, redundant queries simultaneously, right when search traffic was highest.

The Solution

We fixed this with three changes, all scoped around how a rank gets fetched and cached, not around how admin edits it:

• Negative caching — "no rank set for this airline" is now cached as its own explicit result, not left unrepresented. An unranked airline costs exactly one database check, ever, instead of one on every appearance in every search.
• Locking around the retrieval-and-set step — on a cache miss, we acquire a lock before querying the database and writing the result into Redis. Any other request that misses on that same airline while the lock is held simply waits and then reads the value that's now sitting in Redis, instead of independently repeating the database query. This is the piece that actually breaks the stampede: N concurrent misses for the same airline collapse into exactly one database round trip.
• No TTL — cached entries, positive or negative, stay valid indefinitely and are only invalidated when admin actually changes a rank. Without a timer, there's no moment where a batch of entries expires together and re-triggers a herd later — the only path to a database call is a genuine first-time miss, and that path is already protected by the lock.

The Results

• Database calls for airline rank collapsed to a true first-time-only event per airline — after that, every lookup, ranked or confirmed-unranked, is served from Redis for as long as the value stays valid
• Concurrent misses on the same airline now coordinate instead of stampeding, so bursts of search traffic no longer translate into bursts of duplicate database load
• Admin rank changes take effect immediately, since there's no TTL window standing between an update and the cache reflecting it