Kafka architecture consists of several key components:

• Producer: Applications that publish (write) events to Kafka topics
• Consumer: Applications that subscribe to (read) events from topics
• Broker: Kafka servers that store data and serve client requests
• Topic: Categories or feed names to which records are published
• Partition: Sub-divisions of topics for parallelism and scalability
• Zookeeper: Coordinates and manages Kafka cluster metadata (being replaced by KRaft)

A topic is a category or stream name to which producers send messages and from which consumers read messages. Topics are divided into partitions for scalability and parallelism.

Partitioning benefits:

• Enables parallel processing by multiple consumers
• Distributes data across multiple brokers
• Provides ordering guarantees within each partition
• Allows horizontal scaling

Example: A topic "user-events" with 3 partitions can have messages distributed across partitions 0, 1, and 2 based on a key or round-robin.

Zookeeper manages Kafka cluster coordination and metadata:

Responsibilities:

• Broker discovery and health monitoring
• Topic and partition metadata storage
• Leader election for partitions
• Consumer group coordination (legacy)
• Configuration management

Note: Kafka is moving away from Zookeeper dependency with KRaft (Kafka Raft) mode, which handles metadata management internally.

Kafka Headers are optional metadata key-value pairs attached to each message record.

Use cases:

• Routing: Route messages based on headers
• Tracing: Add correlation IDs for distributed tracing
• Security: Include authentication tokens
• Content metadata: MIME types, encoding information
• Source identification: Origin system information

// Producer adding headers
ProducerRecord<String, String> record = new ProducerRecord<>(
    "my-topic", "key", "value");
record.headers().add("correlation-id", "12345".getBytes());
record.headers().add("source-system", "order-service".getBytes());

Benefits: Headers don't affect partitioning and allow metadata without modifying message payload.

Kafka supports two cleanup policies:

1. Delete policy (cleanup.policy=delete):

• Deletes old log segments based on time/size
• log.retention.hours=168 (7 days default)
• log.retention.bytes=-1 (unlimited size default)
• log.segment.bytes=1GB (segment size)

2. Compact policy (cleanup.policy=compact):

• Keeps the latest value for each key
• Useful for changelog topics
• Background compaction process
• Maintains ordering within partitions

Combined policy:

cleanup.policy=compact,delete
# Both compaction and time-based deletion

Monitoring: Track log size and compaction metrics to ensure proper cleanup.