Raft Log Compaction and Snapshots, Explained for Interviews

Raft keeps a replicated log of commands. Left alone, that log grows forever, restarts replay it from the beginning, and new followers must download all of it. Log compaction through snapshots is the standard fix, and it's a common follow-up once you've explained leader election and log replication.

What a snapshot contains

• The state machine's state after applying every entry up to some index.
• lastIncludedIndex and lastIncludedTerm: the index and term of the last entry the snapshot replaces, kept so the AppendEntries consistency check still works for the next entry.
• The latest cluster configuration as of that index, so membership changes aren't lost.

After snapshotting, the server discards log entries up to lastIncludedIndex.

Each server snapshots independently

Snapshots don't go through consensus. Each node compacts its own committed entries when its log passes a size threshold. That keeps the leader simple and avoids shipping big state over the network in the common case.

InstallSnapshot: when a follower is too far behind

If the leader has already discarded the entries a lagging follower needs, it sends its snapshot with the InstallSnapshot RPC, in chunks. The follower:

1. Discards any log entries covered by the snapshot (keeping later entries if its log already contains the snapshot's last entry with a matching term).
2. Resets its state machine from the snapshot.
3. Continues normal replication from lastIncludedIndex + 1.

Trade-offs worth volunteering

• Snapshot frequency: too often wastes I/O; too rarely means long restarts and big logs. A size-based threshold is typical.
• Not blocking writes: use copy-on-write data structures or fork(), as Redis does for RDB, so the state machine keeps serving while a snapshot is written.
• Alternatives: LSM-style incremental compaction or log cleaning (as in LogCabin's design discussion) can avoid full-state snapshots for very large state.

Index arithmetic pitfalls

Once the log has a prefix removed, the array index no longer equals the Raft index. Store an offset (lastIncludedIndex) and convert everywhere, including the prevLogIndex and prevLogTerm lookups. This is where most implementations break.

One-line summary: "Each node snapshots its committed state and truncates the log; the leader sends InstallSnapshot to followers that fall behind the truncated prefix."

Disclosure: written by the Linkjob AI interview assistant team. Free to use for study.