Apache Kafka

Kafka 4.x in KRaft mode Checked Nov 2026 66 commands

The model

A topic is split into partitions. Each partition is an append-only log, and each record gets an offset, its number in that log.
Order holds only inside one partition. A record's key picks its partition, so records with one key stay in order.
Reading deletes nothing. Each consumer group keeps its own offsets, so groups read independently and can replay until retention removes the data.
Within a group, each partition goes to one consumer. So the partition count caps how many consumers work in parallel.
Each partition has one leader and some followers. A write is committed once every in-sync replica (ISR) has it, and consumers see only committed records.

WRITE arrows carry a record to every in-sync replica. READ arrows show a consumer pulling from the leader and saving its place.

Topics & partitions

where every record lives

The parts of a cluster

Write recordsA producer. It also picks each record's partition.

Store and serve recordsA broker, which is a Kafka server.

Read recordsA consumer, in order per partition.

Keep the metadataA controller.

TrapKafka 4.x has no ZooKeeper. It runs only in KRaft mode.

Asked in interviews

AskedExplain the core components of Kafka architecture.

How a record finds its partition

Send a keyed recordThe key's hash picks the partition, so one key stays in one partition.

Send a keyless recordIt goes to a sticky partition, which changes after at least batch.size bytes.

TrapAdding partitions can move a key to another partition.

Asked in interviews

AskedWhat is a Kafka topic and how does partitioning work?

Choose the partition count

Size for throughputUse max(t/p, t/c), with target t and one partition's producer rate p and consumer rate c.

Size for consumers12 consumers at peak need at least 12 partitions.

TipEach partition must fit on one broker. Very high counts cost file handles, metadata and recovery time.

TrapYou can never reduce the count. Raising it remaps keys, which breaks per-key order for existing keys.

Asked in interviews

AskedHow do you choose the number of partitions for a Kafka topic?

Kafka, queues and Pulsar

Replay old messages, or serve many readersUse Kafka. It keeps messages after a read, while a traditional queue deletes them.

Get one order across a whole streamA traditional queue can. Kafka orders only within a partition.

Scale serving and storage separatelyUse Pulsar. It keeps data in Apache BookKeeper, so a new broker serves traffic at once.

Get native tenants and built-in geo-replicationUse Pulsar. Kafka uses MirrorMaker 2, a separate tool.

TrapKafka never pushes records. Consumers pull them.

Asked in interviews

AskedWhat is the difference between Kafka and traditional message queues?AskedWhat is the difference between Kafka and Apache Pulsar?

Manage topics from the shell

List topicskafka-topics.sh --bootstrap-server localhost:9092 --list

Create a topickafka-topics.sh --create --topic t --partitions 6 --replication-factor 3

See a topic's leaders, replicas and ISRkafka-topics.sh --describe --topic t

Read a topic from the startkafka-console-consumer.sh --topic t --from-beginning

Raise the partition countkafka-topics.sh --alter --topic t --partitions 12

Delete a topickafka-topics.sh --delete --topic t

TipEvery tool needs --bootstrap-server. The other rows on this sheet leave it out to stay short.

Asked in interviews

AskedWhat is a Kafka topic and how does partitioning work?

Add headers to a record

Carry a trace ID or content type outside the payloadrecord.headers().add("trace-id", bytes)

TipHeaders don't affect partitioning.

Asked in interviews

AskedWhat are Kafka Headers and when would you use them?

Producer settings

trade durability, order and speed

Choose acks

acks=0Send without waiting for any broker

acks=1Wait for the leader only

acks=allWait for every in-sync replica (default)

must have the recordnot waited for

Trapacks=1 turns off default idempotence. A record is lost if the leader fails before the followers copy it.

Asked in interviews

AskedExplain Kafka producers and their key configurations.

Batch and retry

Wait to fill a batch, 5 ms by default since 4.0linger.ms=5

Cap a batch's bytes per partitionbatch.size=65536

Limit total retry time, as retries are unlimited by defaultdelivery.timeout.ms=120000

Trapmax.in.flight.requests.per.connection above 1 can reorder retried batches when idempotence is off.

Asked in interviews

AskedExplain Kafka producers and their key configurations.

Pick a compression codec

Favour speed for high throughputcompression.type=lz4

Favour compression when network or storage is costlycompression.type=zstdcompression.type=gzipgzip uses more CPU.

Balance compression and speedcompression.type=snappy

TrapIf a topic sets a different compression.type, the broker recompresses every batch. That costs CPU.

Asked in interviews

AskedExplain Kafka's compression and its trade-offs.

Consumer groups & offsets

read in parallel, resume after a crash

Share work in a group

Join a groupgroup.id=billing

Let two apps read a topicGive each its own group.id, so each keeps its own offsets.

Add a 5th consumer to 4 partitionsIt sits idle. Each partition goes to one consumer in a group.

TrapTwo apps in one group each read only part of the topic.

Asked in interviews

AskedHow do Kafka consumers work and what is a consumer group?

Poll and fetch

Get records in your codeconsumer.poll(Duration.ofMillis(100))

Cap the records one poll() returnsmax.poll.records=500

Make the broker wait to fill a fetchfetch.min.bytes=50000fetch.max.wait.ms=500

TipA fetch is the network request under poll(). KafkaConsumer has no public fetch() method.

Trapmax.poll.records limits one poll(), not fetching. The consumer caches fetched records.

Asked in interviews

AskedWhat is the difference between poll and fetch in a Kafka consumer?

Commit offsets

Commit every 5 seconds (default)auto.commit.interval.ms=5000

Commit from your own codeconsumer.commitSync()consumer.commitAsync()Set enable.auto.commit=false first.

Choose where a group with no commit startsauto.offset.reset=earliestThe default is latest.

Reset a group's offsetskafka-consumer-groups.sh --reset-offsets --to-latest --group g --topic tThis previews. Add --execute with the group stopped. --to-earliest replays.

TipYou commit the offset of the next record to read.

TrapCommit before processing and a crash loses messages. Commit after processing and a crash repeats them.

Asked in interviews

AskedWhat is offset management in Kafka?

The offsets topic

Find where committed offsets live__consumer_offsetsA compacted internal topic, holding the latest offset per group and partition.

Expire an empty group's offsetsoffsets.retention.minutes=10080After 7 days by default.

TrapNever add partitions to this topic. Each group maps to one partition, and a new count breaks that mapping.

Asked in interviews

AskedWhat is the purpose of the __consumer_offsets topic?

Rebalance a group

Stop the whole group (eager)RangeAssignor

Move only what must move (cooperative)CooperativeStickyAssignor

Let the broker assign partitions (4.0+)group.protocol=consumerNo member has to wait for the others to rejoin.

Restart a consumer without a rebalancegroup.instance.id=app-1It must return before the session timeout.

TipA rebalance starts when a member joins, leaves or crashes, or when a subscribed topic gains partitions.

TrapA consumer past max.poll.interval.ms between polls leaves the group, even with live heartbeats.

Asked in interviews

AskedWhat triggers a consumer group rebalance, and how do the rebalance protocols differ?

Find and fix consumer lag

Measure lagkafka-consumer-groups.sh --describe --group g

Catch up when consumers are too slowAdd consumers up to one per partition, then add partitions. Or process on another thread.

Fix one partition that lags aloneLook for a hot key, one that gets too much traffic.

TipLag is the latest offset minus the consumer's current offset. The records-lag-max metric reports it.

Asked in interviews

AskedWhat causes consumer lag and how do you address it?

Handle a poison message

Handle a transient errorRetry with backoff, or send the record to a retry topic.

Handle a record that always failsSend it to a dead letter topic with error headers, then commit.

Skip a record that won't deserializeCatch RecordDeserializationException and seek past its offset.

TipConsumer groups have no built-in dead letter queue, so you build it.

TrapSkipping breaks per-key order. When order matters, pause the partition and alert.

Asked in interviews

AskedHow do you handle poison messages in a Kafka consumer?

Replication & durability

how a write survives a broker failure

Replicas and the ISR

Set how many copies a partition hasUse --replication-factor 3 at creation. The count includes the leader.

Replace a failed leaderThe controller elects one from the ISR.

Keep a follower in the ISRreplica.lag.time.max.ms=30000It must heartbeat and reach the log end within this time.

Elect a safe replica outside the ISR (4.0+)Eligible Leader Replicas (ELR). On by default for new clusters since 4.1.

TipKafka measures follower lag only in time. The old replica.lag.max.messages setting no longer exists.

Asked in interviews

AskedExplain Kafka's replication mechanism.AskedExplain Kafka's ISR (In-Sync Replica) mechanism.

Keep committed writes safe

Records up to the high watermark are on every in-sync replica, and consumers can read them. A leader failure can lose the rest.

Use the common setup, with replication factor 3min.insync.replicas=2acks=all

Let an out-of-sync replica become leaderunclean.leader.election.enable=trueOff by default. It can lose committed data.

TrapIf the ISR drops below min.insync.replicas, acks=all writes fail with NotEnoughReplicas. Strong consistency costs availability.

Asked in interviews

AskedWhat are Kafka's consistency guarantees?

Delivery guarantees

lost, repeated or exactly once

Choose a delivery semantic

Lose messages, never repeat themacks=0At-most-once. Commit before processing.

Never lose messages, maybe repeat themacks=allAt-least-once, the default. Commit after processing.

Make each message take effect oncetransactional.id=app-1Exactly-once, with read_committed consumers.

TrapExactly-once covers reads and writes on Kafka topics. A write to an outside database needs its own idempotent handling.

Asked in interviews

AskedWhat are Kafka's delivery semantics?

Exactly-once and transactions

producer.initTransactions();registers the IDproducer.beginTransaction();producer.send(record);producer.sendOffsetsToTransaction(offsets, meta);producer.commitTransaction();commits writes and offsets

Drop batches that a retry sends twiceenable.idempotence=trueThe broker checks a producer ID and a sequence number.

Turn on exactly-once in Kafka Streamsprocessing.guarantee=exactly_once_v2

Stop an old instance after a restartStart with the same transactional.id. Brokers reject the old instance's writes and commits.

Tipread_committed hides aborted and unfinished records. The default, read_uncommitted, returns everything.

TrapA fenced producer can't abort, so close it on a fatal error. On any other error, abort and retry.

Asked in interviews

AskedExplain Kafka's exactly-once semantics implementation.AskedWhat are Kafka transactions and how do they work?

Stop duplicates

Find where duplicates come fromProducer retries, lost acknowledgements and rebalances.

Deduplicate in the consumerUse the message ID as a primary key, and write with upserts.

Deduplicate a Kafka-to-Kafka pipelineUse transactions with read_committed consumers.

TipThe idempotent producer, on by default since 3.0, stops the duplicates that retries cause.

TrapAn in-memory set of seen IDs works only in one process, and a restart loses it.

Asked in interviews

AskedHow do you handle duplicate messages in Kafka?

Retention & compaction

what Kafka keeps, and for how long

Set retention

Delete old segments after 7 days (default)log.retention.hours=168

Also limit size per partition, where -1 is no limitlog.retention.bytes=10737418240

Compact and delete by age togethercleanup.policy=compact,delete

Tipdelete is the default cleanup.policy. A segment is 1 GB by default.

TrapRetention removes whole segments by age or size, whether or not anyone has read them.

Asked in interviews

AskedWhat are Kafka's log retention and cleanup policies?

Compact a log

beforeFive records. A letter is a key, and a number is its version.

afterThe latest value per key stays, at its original offset.

Keep the latest value per keycleanup.policy=compact

Delete a keydelete.retention.ms=86400000Send a tombstone, that key with a null value. The cleaner removes it after this time.

TipUse it for changelog topics and Streams state stores.

TrapThe cleaner never compacts the active segment, so older values of a key can stay readable for a while.

Asked in interviews

AskedExplain Kafka's log compaction feature.

Tier old data to remote storage (3.9+)

Turn it on for brokers, then per topicremote.log.storage.system.enable=trueremote.storage.enable=true

Set how long segments stay on local disklocal.retention.ms=86400000retention.ms still sets the total.

TipLong retention gets cheaper, and storage grows without new brokers.

TrapKafka ships no remote storage implementation, so you plug one in. Compacted topics aren't supported.

Asked in interviews

AskedWhat is tiered storage in Kafka, and what problem does it solve?

Controllers & KRaft

who keeps the metadata

Controllers and KRaft

Recall what ZooKeeper didIt held metadata, tracked brokers and elected the controller.

Name what replaced ZooKeeper in Kafka 4.0KRaft. Controllers keep metadata in a log inside Kafka and agree through Raft.

List the controller's jobsElect leaders, fence silent brokers and track ISR changes.

Explain why failover is fastOne controller is active. The others hold the same log as hot standbys.

TrapA cluster still in ZooKeeper mode must migrate to KRaft before it upgrades to 4.x.

Asked in interviews

AskedWhat was ZooKeeper's role in Kafka, and what replaced it?AskedWhat is the role of the Kafka Controller?

Survive a network partition

Stop two sides deciding at once (split-brain)Only a majority of controllers can decide. 3 survive 1 failure, and 5 survive 2.

Remove an isolated leaderA silent broker is fenced after broker.session.timeout.ms, and in-sync replicas take over.

Reject an old leader's commandsEach new controller and leader gets a higher epoch. Brokers reject older ones.

TipKafka favours consistency. It may stop taking writes during a network partition.

Asked in interviews

AskedHow does Kafka handle network partitions and split-brain scenarios?

Upgrade without breaking clients

Let old and new versions talkThe client uses the highest API version both sides know.

Upgrade a cluster with no downtimeUpdate brokers one at a time. Run kafka-features.sh only after all are done.

TrapKafka 4.0 removed old protocol versions. 4.x Java clients and brokers need the other side on 2.1 or later.

Asked in interviews

AskedWhat is Kafka's approach to backward compatibility?

Speed & monitoring

why it's fast, and how to see trouble

Why Kafka is fast

Explain fast writesAppend-only logs make writes sequential.

Explain fast readsThe OS page cache serves recent data. Zero-copy sendfile sends it straight to the network.

TipBatches and compression shrink each request. A fully committed write takes about 10 ms.

TrapTLS stops zero-copy, because encryption runs in user space.

Asked in interviews

AskedHow does Kafka ensure high throughput?

Tune for throughput or latency

Favour throughput on the producerbatch.size=65536linger.ms=20compression.type=lz4

Favour the lowest latencybatch.size=0linger.ms=0

Clear request queues on a brokernum.network.threads=8num.io.threads=16

Prepare the OSRaise the file descriptor limit. Every log segment holds an open file.

Asked in interviews

AskedHow do you tune Kafka for maximum performance?

Monitor a cluster

Alert above 0, when followers fall behindUnderReplicatedPartitions

Alert above 0, when partitions have no leaderOfflinePartitionsCount

Check that exactly one node reports 1ActiveControllerCount

Spot saturated request threads, below about 0.3RequestHandlerAvgIdlePercent

TipBrokers and clients expose metrics through JMX.

Asked in interviews

AskedHow do you monitor Kafka cluster performance?

Avoid the common mistakes

Share a producerUse one thread-safe producer, never a new one per message.

Catch failed sendsHandle exceptions, or you lose records without notice.

Look up records by keyUse a database. A topic isn't built for lookups.

Set up productionRun more than one broker, with replication factor 3.

TrapDon't rely on topic auto-creation. It applies broker defaults.

Asked in interviews

AskedWhat are some common Kafka anti-patterns to avoid?

Security

prove who, decide what, encrypt the data

Authenticate and authorize

Let a user read a topickafka-acls.sh --add --allow-principal User:a --operation Read --topic t

TrapOnce authorizer.class.name is set, a resource with no ACLs is open only to super users.

Asked in interviews

AskedExplain Kafka's security features.

Encrypt data

Encrypt data on the networksecurity.protocol=SSL

Encrypt data on diskKafka has no built-in support. Use dm-crypt or cloud disk encryption.

Hide fields from Kafka itselfEncrypt them in the application before producing.

TipA client adds a truststore to verify the broker, and a keystore only for mutual TLS.

TrapDisk encryption protects a stolen disk. Any client allowed to read the topic still sees plain data.

Asked in interviews

AskedWhat is the difference between at-rest and in-transit encryption in Kafka?

Streams, Connect & patterns

tools and designs built on Kafka

Process topics with Kafka Streams

Filter one topic into anotherbuilder.stream("in").filter(f).to("out")

Join or aggregate with stateStreams keeps local state stores. Interactive queries let other code read them.

TipKafka Streams is a library inside your Java or Scala app, so it needs no separate cluster.

Asked in interviews

AskedWhat is Kafka Streams and when would you use it?

Connect and MirrorMaker

Move data with no codeA source connector imports, and a sink connector exports.

Share load across workersUse distributed mode. Workers keep offsets and configs in Kafka topics.

Copy topics between clustersMirrorMaker 2. orders from us-west becomes us-west.orders.

Move a group across clustersCheckpoints translate its offsets.

TipMirrorMaker 2 runs on Connect. Use it for disaster recovery, data locality, migration and compliance.

TrapKafka 4.0 removed the original MirrorMaker.

Asked in interviews

AskedExplain Kafka Connect and its purpose.AskedWhat is Kafka MirrorMaker and its use cases?

Evolve a schema

Let a new schema read old data (default)Backward. Upgrade consumers first.

Let an old schema read new dataForward. Upgrade producers first.

Allow both, or check every earlier versionFull does both. Transitive checks all earlier versions.

TipA Schema Registry enforces the rule. Add only optional fields, with defaults.

TrapDon't remove required fields.

Asked in interviews

AskedHow do you handle schema evolution in Kafka?

Event sourcing and CQRS

Store every state changeWrite an immutable event, keyed by aggregate ID, so one entity's events stay in order.

Keep the full historyretention.ms=-1

Load one aggregate's current stateBuild it in a Kafka Streams state store. Kafka reads by offset, not by key.

Serve queries with CQRSThe command side produces events, and the query side turns them into read models.

TipA CQRS read just after a write may not show it, because the query side is eventually consistent.

TrapDon't compact the event topic. Put snapshots in a separate compacted topic.

Asked in interviews

AskedHow do you implement event sourcing with Kafka?AskedHow do you implement CQRS (Command Query Responsibility Segregation) with Kafka?

Pro · $10/mo

This sheet stays free. With Pro, you can:

Every answer on this sheet. Read the full answer to all 41 questions it links, not just the 5 free ones.
My sheet. The cards you star and the ones you miss in Quiz me, from every sheet, on one page to quiz and print.
PDFs. Download this sheet and My sheet, laid out for A4, with every question a link to its answer.
The rest of Pro. Every answer in the question library, and 1,000 AI credits a month.

Get Pro or browse more free sheets

The model

Topics & partitions

The parts of a cluster

How a record finds its partition

Choose the partition count

Kafka, queues and Pulsar

Manage topics from the shell

Add headers to a record

Producer settings

Choose acks

Batch and retry

Pick a compression codec

Consumer groups & offsets

Share work in a group

Poll and fetch

Commit offsets

The offsets topic

Rebalance a group

Find and fix consumer lag

Handle a poison message

Replication & durability

Replicas and the ISR

Keep committed writes safe

Delivery guarantees

Choose a delivery semantic

Exactly-once and transactions

Stop duplicates

Retention & compaction

Set retention

Compact a log

Tier old data to remote storage (3.9+)

Controllers & KRaft

Controllers and KRaft

Survive a network partition

Upgrade without breaking clients

Speed & monitoring

Why Kafka is fast

Tune for throughput or latency

Monitor a cluster

Avoid the common mistakes

Security

Authenticate and authorize

Encrypt data

Streams, Connect & patterns

Process topics with Kafka Streams

Connect and MirrorMaker

Evolve a schema

Event sourcing and CQRS

Backend Development

ASP.NET Core

C#

C++

Django

This sheet stays free. With Pro, you can: