Kafka has two cleanup policies, set per topic with cleanup.policy: delete, the default, and compact.
Delete removes whole old log segments once they pass a time or size limit:
log.retention.hours=168 keeps data for 7 days by default.
log.retention.bytes=-1 sets no size limit by default, and any limit applies per partition.
log.segment.bytes=1073741824 sets the segment size to 1 GB.
Compact keeps the latest value for each key, which suits changelog topics. A background process compacts the log and keeps the order of records within each partition.
Both can run together.
cleanup.policy=compact,delete
# Both compaction and time-based deletion
Track log size and compaction metrics to make sure cleanup keeps up.
Rewriting in plainer words…
This answer doesn't lend itself to a diagram - it reads best . No credits were charged.
