学习资料

1.https://thehoard.blog/how-kafkas-storage-internals-work-3a29b02e026

2.https://www.conduktor.io/understanding-kafkas-internal-storage-and-log-retention

整体设计

Role of Indexing within the Partition

How these entries are added inside the index file is defined by thelog.index.interval.bytesparameter, which is 4096 bytes by default. This means that after every 4096 bytes added to the log, an entry gets added to the index file. Suppose the producer is sending records of 100 bytes each to a Kafka topic. In this case, a new index entry will be added to the .index file after every 41 records (41*100 = 4100 bytes) appended to the log file.

基于offset找消息

基于时间找消息

Rolling segments

Maximum segment size - configured bylog.segment.bytes, defaults to 1 Gb

Rolling segment time - configured bylog.roll.msandlog.roll.hours, defaults to 7 days

Index/timeindex is full - The index and timeindex share the same maximum size, which is defined by the** log.index.size.max.bytes**, defaults to 10 MB