Apache Kafka.

Kafka's architecture has producers that write events, brokers that store them and consumers that read them, while controllers manage the cluster's metadata. Kafka also calls an event a record.

• Producers are applications that publish, or write, events and choose each event's partition.
• Consumers are applications that subscribe to topics and read events in order within each partition.
• Brokers are the Kafka servers that store events on disk and serve client requests.
• Topics are named feeds, such as payments, that producers publish to.
• Partitions split each topic into ordered logs across brokers, for parallelism and scalability.
• Controllers keep the metadata, such as which topics exist and where each partition lives.

Older clusters kept this metadata in ZooKeeper, a separate coordination service. Since Kafka 4.0, clusters run only in KRaft mode. The controllers agree on each change through Raft, a protocol in which a majority must accept it.

A topic is a named stream that producers send messages to and consumers read from. Kafka calls each message a record, and divides each topic into partitions, which are ordered logs spread across brokers.

The producer picks each record's partition. A keyed record goes to a partition chosen by hashing its key, so one key maps to one partition while the partition count is unchanged. A record without a key goes to a sticky partition, which changes once at least batch.size bytes have gone to it.

Partitions bring these benefits:

• Several consumers can read a topic in parallel.
• Records spread across brokers.
• Order holds within each partition, though not across partitions.
• A topic scales out as you add partitions and brokers.

For example, a topic user-events with 3 partitions stores records in partitions 0, 1 and 2.

ZooKeeper was the external service that held Kafka's cluster metadata, until Kafka 4.0 removed it in favour of KRaft mode.

In ZooKeeper mode it handled:

• Broker discovery and health monitoring
• Topic, partition and configuration metadata
• Electing the controller broker, which then chose partition leaders
• Consumer group coordination for the legacy consumer

In KRaft mode, a few controller nodes keep this metadata in a log inside Kafka. They agree on each change with Raft, a consensus protocol where a majority must accept it. Operators run one system instead of two. A Kafka cluster still in ZooKeeper mode must migrate to KRaft before upgrading to 4.x.

Kafka headers are optional key-value pairs of metadata attached to each record, next to its key and value. Use them for information about a message that should not live inside the payload.

Common uses:

• Routing, where consumers route messages based on a header.
• Tracing, where a correlation ID follows a request across services.
• Security context, such as an authentication token.
• Content metadata, such as the MIME type or encoding.
• Source identification, naming the origin system.

// Producer adding headers
ProducerRecord<String, String> record = new ProducerRecord<>(
    "my-topic", "key", "value");
record.headers().add("correlation-id", "12345".getBytes());
record.headers().add("source-system", "order-service".getBytes());

Headers do not affect partitioning, and they add metadata without changing the message payload.

Kafka has two cleanup policies, set per topic with cleanup.policy: delete, the default, and compact.

Delete removes whole old log segments once they pass a time or size limit:

• log.retention.hours=168 keeps data for 7 days by default.
• log.retention.bytes=-1 sets no size limit by default, and any limit applies per partition.
• log.segment.bytes=1073741824 sets the segment size to 1 GB.

Compact keeps the latest value for each key, which suits changelog topics. A background process compacts the log and keeps the order of records within each partition.

Both can run together.

cleanup.policy=compact,delete
# Both compaction and time-based deletion

Track log size and compaction metrics to make sure cleanup keeps up.