What is a segmented log?

Logging is essential for recovery to ensure fault tolerance in the client-server model, where the server can fail anytime. Keeping the log in a single file (as discussed in the write-ahead log) will make it difficult for the server to go through the log file from the start of the log file until the last commit (the point from where it has to start), which is also time-consuming. Time consumption is not our only problem with it. In that given time period, the server will respond to the client with outdated data, which affects the consistency of the system.

Definition

Managing the log file mentioned above (containing all its operations) can be chaotic, so we need to divide the log file after a specific length. The process of creating segments of the log file is called a segmented log. Such segmented log files are easier to process and manage.

Solution

The following two operations are performed on the log files:

  1. Write: In the write operation, the log file is split into segments when the file reaches a specific size. Because of different files, we need to map these files with sequence numbers according to the order of the operations. The filename of each segment is generated by using the following two steps:

    a. The sequence number of each segment should be the filename and transaction offset (transaction ID).

    sequence_number = filename + " - " + transaction_offset;

    b. The name of each segment should be a specific prefix and sequence number of the log.

    segmented_log_filename = prefix + " - " + sequence_number;

Note: We use transaction_offset in segmented_log_filename. This helps us identify what transaction is in which segmented_log_filename. It is part of the sequence_number, which is used in segmented_log_filename.

  1. Read: In the read operation, all segmented_log_filename are loaded into an array, and we will start reading the logs after we perform the following two steps:

    a. Split each stored segmented_log_filename to get the sequence_number.

    b. Sort all the files according to the sequence_number.

Example

Let’s discuss some examples where a segmented log is used:

  1. Coordination services such as ZooKeeper (learn more about it in Educative’s exclusive course) and RAFT use a segmented log.
  2. Kafka’s (learn more about it in Educative’s exclusive course) implementation of storage follows the segmented log.
  3. Cassandra and all other databases (both SQL and NoSQL) use the segmented log.

New on Educative
Learn to Code
Learn any Language as a beginner
Develop a human edge in an AI powered world and learn to code with AI from our beginner friendly catalog
🏆 Leaderboard
Daily Coding Challenge
Solve a new coding challenge every day and climb the leaderboard

Free Resources

Copyright ©2025 Educative, Inc. All rights reserved