Redis — caching and pub/sub — SQL & Relational Databases

Redis (Remote Dictionary Server) is an in-memory data structure store used primarily as a cache, message broker, and real-time data store. Unlike disk-based databases that persist data to storage and read it back on query, Redis stores all data in RAM, enabling sub-millisecond read and write latency at high throughput (typically 100,000–1,000,000 operations per second on a single node). Redis supports a rich set of native data structures — strings, hashes, lists, sets, sorted sets, streams, and HyperLogLog — each with purpose-built commands optimised for specific access patterns. This makes Redis far more powerful than a simple key-value store.

For data engineers, Redis appears in three distinct roles. As a cache, Redis stores the computed results of expensive database queries with a configured TTL (time to live) — subsequent requests return the cached result in microseconds rather than re-running the query. As a message broker, Redis Streams and the pub/sub system enable decoupled, asynchronous communication between pipeline components — a producer writes events, consumers read and process them independently. As a real-time leaderboard or counter store, Redis sorted sets and atomic increment commands maintain live rankings and counters that update instantly without database round-trips.

Redis persistence options range from fully in-memory (data lost on restart) to semi-durable (RDB snapshots at configured intervals) to near-durable (AOF append-only file logging every write command). For cache use cases, no persistence is needed — cache misses simply refill from the source. For message broker use cases where message loss is unacceptable, AOF with fsync on every write (appendfsync always) provides near-PostgreSQL durability at the cost of write throughput. Understanding the persistence trade-offs determines whether Redis is appropriate for each specific use case in a data pipeline.

Analogy🏏Cricket

🏏 Think of it like cricket: A SELECT query is precisely how a selection committee picks a playing XI. FROM is the full list of centrally contracted players — the raw pool. WHERE is the fitness and eligibility screen: injured or unavailable players are removed before anyone debates merit, and the fewer names that survive this screen, the faster the meeting goes — exactly why a good WHERE clause matters more than anything downstream. ORDER BY is ranking the survivors by recent form, then by experience as the tiebreaker. LIMIT 11 takes the top of that ranked list and stops. The committee never ranks the entire national player pool and then discards thousands of names — and neither should your query force the database to sort millions of rows it will immediately throw away. The order of operations is the whole game: filter first, sort what remains, take only what you need.