How Redis Works Under the Hood

Ask any backend developer why Redis is fast and you'll get "it's in-memory." That's true and it's also not an explanation. Plenty of things are in-memory. What Redis actually bought with that constraint is the absence of a concurrency model, and every operational trade-off you will hit in production — the slow-command stalls, the fork memory spike, the encoding cliffs — falls out of that one decision.

The Mental Model

Redis executes all commands on one thread, sequentially, driven by an event loop. There are no mutexes protecting the keyspace because nothing else touches it — only one command runs at a time, start to finish.

Two caveats so the model is accurate rather than just tidy. Redis has always had background threads for jobs that must not block the loop: fsync, closing large files, and freeing big objects. And since Redis 6 there is io-threads, which can move socket reads, parsing and reply writing off the main thread — but not command execution, and it ships disabled ("By default threading is disabled"). So the single-threaded story holds exactly where it matters: your command is never interleaved with anyone else's.

All three clients connect, but only one command runs at a time. The event loop queues everything else. This is the source of both the performance and the footgun: no slow command can be interrupted.

The Event Loop

Redis implements a reactor pattern on top of OS I/O multiplexing. The main loop looks like this in pseudocode:

while (!stop) {
    // wait for events with timeout = next_timer_fire
    numevents = ae_poll(eventLoop, timeout);
 
    for (i = 0; i < numevents; i++) {
        // handle file event (read from client / write response)
        processFileEvent(eventLoop->fired[i]);
    }
 
    // run any due time events (expiry, BGSAVE triggers, etc.)
    processTimeEvents(eventLoop);
}

ae_poll wraps the best available multiplexer on the current OS: epoll on Linux, kqueue on macOS/BSD, falling back to select. The call returns a list of sockets that are ready — either a client has sent bytes, or a socket buffer has drained enough to accept a response. Redis never blocks waiting for a single socket; it handles whatever is ready, then comes back around.

File events handle all client I/O: accepting connections, reading RESP-encoded commands, writing responses. Time events run periodic tasks: expiring keys, triggering background saves, replication heartbeats, cluster health checks.

Because everything is sequential, there are no locks, no condition variables, and no cache lines bouncing between cores. A GET is a hash lookup and a reply write, and that's the whole code path. If you want a number for your hardware, redis-benchmark -t get -n 100000 will give you one — and run it with and without -P 16 pipelining, because the two answers differ by roughly an order of magnitude and people quote them interchangeably.

The Keyspace: A Dict of Dicts

Every Redis database (there are 16 by default) is a dict — a hash table mapping string keys to Redis objects. The hash table uses separate chaining: each bucket is a linked list of entries that hash to the same slot.

The interesting part is how Redis handles resizing. Rather than pausing while it rehashes a full table, Redis uses incremental rehashing:

Normal state:
  ht[0] → active hash table (size N)
  ht[1] → NULL

During resize:
  ht[0] → old table (being drained)
  ht[1] → new table (size 2N or N/2)
  rehashidx = 47  // current bucket being moved

Every command that touches a key under rehash migrates one bucket from ht[0] to ht[1]. Lookups check both tables. Once ht[0] is empty, ht[1] becomes the new ht[0]. The rehash is spread across thousands of operations — no single command pays the full cost.

Data Structures and Their Hidden Encodings

This is the part most people skip, and it's where real memory efficiency comes from.

Redis has five classic types — String, List, Hash, Set, Sorted Set — and several later ones layered on top of them (Streams are their own structure; Bitmaps, HyperLogLog and Geo are all Strings with special commands). Each of the five has multiple internal encodings that Redis switches between based on size. The encoding is invisible to the client — a HSET works regardless of whether the hash is stored as a listpack or a hashtable — but it matters enormously for memory.

Strings: int, embstr, raw

A Redis string is not a C string. It's an SDS (Simple Dynamic String). There isn't one header type — sds.h defines five (sdshdr5, 8, 16, 32, 64) so that a short string doesn't pay 32-bit length fields. The sdshdr32 variant looks like this:

struct __attribute__ ((__packed__)) sdshdr32 {
    uint32_t len;         // current length — O(1) strlen
    uint32_t alloc;       // capacity, excluding header and null terminator
    unsigned char flags;  // low 3 bits pick which of the 5 header types this is
    char buf[];           // the bytes, binary-safe, plus a null for C interop
};

But before allocating an SDS at all, Redis checks:

• int encoding: if the value is a long integer that fits in a pointer, Redis stores it directly in the object pointer field. No heap allocation. Integers 0–9999 are pre-allocated as shared objects at startup — SET counter 42 returns a pointer to a global singleton.
• embstr encoding: if the value is ≤ 44 bytes, Redis allocates the robj header and the SDS in a single 64-byte malloc call. One allocation, one cache line. Immutable — any modification converts to raw.
• raw encoding: separate allocations for the object header and the SDS. Used for strings longer than 44 bytes.

Run OBJECT ENCODING mykey to see which encoding Redis chose. If your small string keys keep hitting raw, you're paying for unnecessary allocations.

Collections: The Compact Encoding Threshold System

All collection types start in a compact encoding and upgrade to a pointer-based structure when they grow past a threshold:

Listpack stores all entries in a single contiguous memory block. No pointers between entries — each entry encodes its own length inline, so you can walk forward or backward through the block. For a hash with 50 fields, this is dramatically more memory-efficient than 50 separate heap allocations with pointer chains.

Intset goes further: a set of pure integers is stored as a sorted array of 16, 32, or 64-bit integers (chosen to fit the largest member). Lookup is binary search — O(log n) — but for a 100-member integer set, 100 binary search iterations is negligible, and the memory savings over a hashtable are significant.

The catch: once a collection crosses a threshold, Redis upgrades the encoding and never goes back. Deleting entries down below the threshold does not revert to listpack. This is important to know when you're tuning memory.

Sorted Sets: The Skiplist + Hashtable Combination

When a sorted set grows past the listpack threshold, Redis uses two structures simultaneously for the same data:

Be careful about which structure serves which command, because this is where write-ups go wrong. The hashtable gives you ZSCORE in O(1) — it maps member directly to score. The skiplist gives you everything ordered: ZRANGE and ZRANGEBYSCORE at O(log(N)+M), and ZRANK at O(log(N)), not O(1). ZRANK cannot come from the hashtable; a hash tells you a member's score but says nothing about how many members sort below it. Redis's own command reference lists ZSCORE as O(1) and ZRANK as O(log(N)), and the rank support was bolted on by augmenting the skiplist with span counts per level.

Why a skiplist instead of a balanced BST? The cleanest answer is antirez's own, from a Hacker News comment on 2010-03-06 when someone asked him exactly that:

There are a few reasons: 1) They are not very memory intensive. It's up to you basically. Changing parameters about the probability of a node to have a given number of levels will make then less memory intensive than btrees. 2) A sorted set is often target of many ZRANGE or ZREVRANGE operations, that is, traversing the skip list as a linked list. With this operation the cache locality of skip lists is at least as good as with other kind of balanced trees. 3) They are simpler to implement, debug, and so forth. For instance thanks to the skip list simplicity I received a patch (already in Redis master) with augmented skip lists implementing ZRANK in O(log(N)). It required little changes to the code.

That last sentence is the receipt for the complexity note above — the man who accepted the patch describes it as O(log(N)).

The memory cost is real: two structures for the same data. But for sorted sets where you need both point lookups and ordered queries, there's no cleaner split.

Persistence: How Redis Doesn't Lose Your Data (Mostly)

RDB: Snapshots via fork + Copy-on-Write

BGSAVE triggers a snapshot:

1. Redis calls fork()
2. Child process has a copy of all memory mappings (virtually instant — CoW)
3. Child writes dataset to temp .rdb file
4. On completion, atomically renames temp file to dump.rdb
5. Child exits
6. Parent continued serving requests throughout

The copy-on-write trick: after fork(), parent and child share the same physical memory pages. When the parent modifies a page to serve a write command, the OS makes a private copy of that page for the parent — the child still sees the original. The snapshot reflects the dataset state at the moment of fork, regardless of subsequent writes.

The problem: under heavy write load, the parent keeps dirtying pages, forcing CoW copies. Your Redis process can temporarily consume close to 2x its normal memory footprint during a BGSAVE. On a machine where you've allocated 80% of RAM to Redis, this causes the OOM killer to visit. Plan your capacity accordingly.

AOF: Command Log with Three Durability Modes

Instead of snapshots, Append-Only File mode logs every write command:

*3\r\n$3\r\nSET\r\n$4\r\nname\r\n$5\r\nAlice\r\n
*3\r\n$6\r\nEXPIRE\r\n$4\r\nname\r\n$4\r\n3600\r\n

Three appendfsync options control when the kernel flushes the buffer to disk:

everysec is the practical default. The one-second window is acceptable for most applications and has minimal throughput impact since the fsync happens asynchronously in a background thread while the main event loop keeps processing commands.

AOF files grow over time. BGREWRITEAOF (also triggered automatically) forks a child to write a compact representation of the current dataset — equivalent to the minimal set of commands to reconstruct current state. New writes during the rewrite get buffered, then appended once the new file is ready.

Hybrid Persistence — and why "the AOF file" stopped being a file

Redis 4.0 introduced aof-use-rdb-preamble, which lets a rewritten AOF start with an RDB snapshot followed by the commands since. On restart you load the fast binary section and replay only a short tail. It shipped as no in Redis 4.0's redis.conf ("This is currently turned off by default in order to avoid the surprise of a format change") and has been yes since Redis 5.0.

Then 7.0 changed the layout, and a lot of writing about Redis persistence hasn't caught up. There is no longer one AOF file with a preamble. Straight from the docs:

Since Redis 7.0.0, Redis uses a multi part AOF mechanism. That is, the original single AOF file is split into base file (at most one) and incremental files (there may be more than one). The base file represents an initial (RDB or AOF format) snapshot of the data present when the AOF is rewritten. The incremental files contains incremental changes since the last base AOF file was created. All these files are put in a separate directory and are tracked by a manifest file.

That directory is appenddirname, default appendonlydir. Practical consequences: your backup script cannot cp appendonly.aof any more, it has to copy the whole directory — and if it does so mid-rewrite you get an invalid backup, which is why the docs tell you to set auto-aof-rewrite-percentage 0 for the duration. Anything in your runbook that names a single AOF file is describing Redis 6.

appendonlydir/
  appendonly.aof.1.base.rdb    # snapshot at last rewrite
  appendonly.aof.1.incr.aof    # commands since
  appendonly.aof.manifest      # what belongs to the set

Replication

Redis replication is async and uses the same fork-based RDB mechanism for initial sync:

The replication backlog is a ring buffer on the primary (default 1 MB) containing recent write commands. When a replica reconnects after a brief disconnect, it sends its last known replication offset. If that offset is still in the backlog, the primary sends only the missed commands — no RDB transfer needed. If the replica was disconnected too long and the backlog has wrapped, a full sync is required.

The +FULLRESYNC reply carries the primary's replication ID and its current offset, not zero — that offset is where the replica's stream begins, and it's only zero on a primary that has never written anything.

Replication is asynchronous, which has a consequence worth stating bluntly: a primary acknowledges your write before any replica has it, so a primary that dies immediately after replying +OK can lose that write even with replicas healthy. WAIT numreplicas timeout is the escape hatch — you call it after your writes and it blocks until that many replicas have acknowledged the current offset, returning how many actually did. It gives you a durability checkpoint on demand; it does not turn replication synchronous, and it is not a consensus protocol.

Practical Implications

The 44-byte embstr boundary matters. Keep your string values under 44 bytes and they get a single allocation. A common mistake is storing JSON blobs as strings — even a 50-byte JSON value gets a less efficient raw encoding. Consider whether a Hash type with individual fields would be more memory-efficient.

Encoding thresholds are one-way gates. Once a collection exceeds hash-max-listpack-entries (default 128), it's a hashtable forever. If you're doing bulk loads that temporarily exceed the threshold, your collection stays a hashtable even if you delete entries back down. If memory density matters, load data in batches that stay under the threshold.

BGSAVE needs headroom. Size your Redis host for 1.5–2x your working set if you use persistence. Watch the used_memory_rss metric during save operations — RSS will spike due to CoW page copies. On Linux, set vm.overcommit_memory = 1 to prevent fork from failing when Redis's virtual address space appears to exceed available RAM (CoW means it rarely does in practice).

The single thread is your friend until it isn't. Most workloads never saturate the event loop. But if you're running Lua scripts, SORT on unsorted sets, or any command that's O(N) on a large N, you'll see everything else wait. Use Redis's slow log (SLOWLOG GET) regularly. Set slowlog-log-slower-than 10000 (10ms) and review it weekly. Surprises come from application code calling LRANGE mylist 0 -1 on a list that grew from 10 elements to 100,000.

The mental model to hold onto: Redis is a fast sequential processor, not a concurrent one. Design your keyspace so that each operation is small, bounded, and cheap. Everything else follows from that.