What is Kafka's log retention and cleanup policies?

Beginner

Answer

Kafka supports two cleanup policies:

1. Delete policy (cleanup.policy=delete):

  • Deletes old log segments based on time/size
  • log.retention.hours=168 (7 days default)
  • log.retention.bytes=-1 (unlimited size default)
  • log.segment.bytes=1GB (segment size)

2. Compact policy (cleanup.policy=compact):

  • Keeps the latest value for each key
  • Useful for changelog topics
  • Background compaction process
  • Maintains ordering within partitions

Combined policy:

cleanup.policy=compact,delete
# Both compaction and time-based deletion

Monitoring: Track log size and compaction metrics to ensure proper cleanup.