Apache Kafka.
Interview cheat sheet.

Apache Kafka is a distributed streaming platform designed for building real-time data pipelines and streaming applications. This cheatsheet covers core Kafka concepts including topics, partitions, producers, consumers, brokers, replication, offset management, consumer groups, message delivery semantics, performance optimization, monitoring, and integration patterns. Key features include high throughput, horizontal scalability, fault tolerance, durability, and support for both pub-sub messaging and stream processing use cases.

What is Kafka?

• Distributed streaming platform for building real-time data pipelines
• Publish-Subscribe messaging system with high throughput
• Horizontally scalable, fault-tolerant, and durable

Key Components

Producer → Kafka Cluster → Consumer
           ├── Broker 1
           ├── Broker 2
           └── Broker N

Topic

• Logical channel for organizing messages
• Partitioned for parallelism and scalability
• Immutable append-only log

Partition

• Ordered sequence of messages
• Unit of parallelism in Kafka
• Each partition has a leader and replicas

Broker

• Kafka server that stores data
• Manages partitions and handles requests
• Multiple brokers form a cluster

Producer

• Publishes messages to topics
• Can specify partition or use partitioner
• Supports async and sync modes

Consumer

• Reads messages from topics
• Part of a consumer group
• Maintains offset for tracking

ZooKeeper (Legacy) / KRaft (New)

• Coordination service for cluster management
• Stores metadata, manages leader election
• KRaft mode eliminates ZooKeeper dependency

Consumer Groups

• Load balancing: Partitions distributed among consumers
• Fault tolerance: Automatic rebalancing on failure
• Scalability: Max consumers = number of partitions

// Consumer group example
Properties props = new Properties();
props.put("group.id", "my-consumer-group");
props.put("enable.auto.commit", "true");

Offset Management

• Current offset: Next message to read
• Committed offset: Last processed message
• Log end offset: Last message in partition

Replication

• Replication factor: Number of copies
• ISR (In-Sync Replicas): Replicas caught up with leader
• Leader election: Automatic on failure

Message Delivery Semantics

1. At most once: Messages may be lost
2. At least once: Messages may be duplicated
3. Exactly once: No loss, no duplication (idempotent producer)

Key Properties

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

// Performance tuning
props.put("batch.size", 16384);
props.put("linger.ms", 1);
props.put("buffer.memory", 33554432);

// Reliability
props.put("acks", "all"); // 0, 1, or all
props.put("retries", 3);
props.put("enable.idempotence", true);

Producer Example

Producer<String, String> producer = new KafkaProducer<>(props);
ProducerRecord<String, String> record = 
    new ProducerRecord<>("topic", "key", "value");
    
// Async send
producer.send(record, (metadata, exception) -> {
    if (exception != null) {
        exception.printStackTrace();
    }
});

// Sync send
try {
    RecordMetadata metadata = producer.send(record).get();
} catch (Exception e) {
    e.printStackTrace();
}

Consumer Properties

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "test-group");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

// Offset management
props.put("enable.auto.commit", "false");
props.put("auto.offset.reset", "earliest"); // earliest, latest, none

// Performance
props.put("max.poll.records", 500);
props.put("fetch.min.bytes", 1);

Consumer Example

Consumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Arrays.asList("topic1", "topic2"));

while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    for (ConsumerRecord<String, String> record : records) {
        // Process record
        System.out.printf("offset = %d, key = %s, value = %s%n",
            record.offset(), record.key(), record.value());
    }
    consumer.commitSync(); // Manual commit
}

Kafka Streams

• Stream processing library
• Stateful transformations
• Exactly-once processing

StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> source = builder.stream("input-topic");
source.filter((key, value) -> value.contains("important"))
      .to("output-topic");

Kafka Connect

• Import/Export data between Kafka and external systems
• Source connectors: Import data into Kafka
• Sink connectors: Export data from Kafka

Schema Registry

• Centralized schema management
• Schema evolution support
• Compatibility checking

Producer Optimization

• Batching: Increase batch.size and linger.ms
• Compression: Use compression.type (gzip, snappy, lz4, zstd)
• Partitioning: Custom partitioner for better distribution

Consumer Optimization

• Parallel processing: Multiple consumers in group
• Fetch size: Tune fetch.min.bytes and max.poll.records
• Session timeout: Balance session.timeout.ms

Broker Optimization

• Replication: Balance between durability and performance
• Log segment: Tune log.segment.bytes
• Memory: Increase heap size and page cache

Message Delivery Guarantees

Producer Acknowledgment Settings

Consumer Group vs Topic Subscription

Partitioning Strategies

Kafka vs Other Messaging Systems

Consumer Rebalancing Triggers

Offset Management Patterns

Performance Optimization Strategies

Common Troubleshooting Scenarios

Monitoring Key Metrics

Kafka Ecosystem Components

Deployment Patterns

Security Configuration

Topic Design

• Naming: Use descriptive, hierarchical names
• Partitions: Start conservative, increase as needed
• Retention: Set based on use case

Error Handling

// Producer error handling
producer.send(record, (metadata, exception) -> {
    if (exception != null) {
        // Log error, retry, or send to DLQ
        logger.error("Failed to send message", exception);
    }
});

// Consumer error handling
try {
    processRecord(record);
    consumer.commitSync();
} catch (Exception e) {
    // Skip bad record or retry
    logger.error("Failed to process record", e);
}

Monitoring

• Metrics to track:
  • Producer: Send rate, error rate, batch size
  • Consumer: Lag, fetch rate, commit rate
  • Broker: Under-replicated partitions, ISR shrinks

Essential Kafka CLI Commands

Configuration Quick Reference

Performance Tuning Checklist

• Producer Batching: Configure batch.size and linger.ms appropriately
• Compression: Enable compression (lz4, snappy, gzip, zstd)
• Partitioning: Choose optimal partition count (start with 2-3x broker count)
• Replication Factor: Balance between durability (3) and performance (1-2)
• Memory Allocation: Set appropriate JVM heap size (6-8GB typical)
• Disk Configuration: Use dedicated disks, consider RAID 10
• Network: Ensure adequate network bandwidth
• Monitoring: Set up comprehensive monitoring and alerting

Topic Design Best Practices

Error Handling Patterns

// Producer with retry and error handling
Properties props = new Properties();
props.put("retries", 3);
props.put("retry.backoff.ms", 1000);
props.put("enable.idempotence", true);

producer.send(record, (metadata, exception) -> {
    if (exception != null) {
        if (exception instanceof RetriableException) {
            // Will be retried automatically
            logger.warn("Retriable error occurred", exception);
        } else {
            // Send to dead letter queue or alert
            logger.error("Non-retriable error", exception);
            sendToDeadLetterQueue(record);
        }
    }
});

// Consumer with error handling
while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    for (ConsumerRecord<String, String> record : records) {
        try {
            processMessage(record);
        } catch (TemporaryException e) {
            // Retry logic
            retryMessage(record);
        } catch (PermanentException e) {
            // Skip or send to DLQ
            logger.error("Permanent error processing message", e);
            sendToDeadLetterQueue(record);
        }
    }
    consumer.commitSync();
}

Monitoring and Alerting Strategy

Common Architecture Patterns

Security Implementation Checklist

• Enable SSL/TLS: Encrypt data in transit
• Configure SASL: Implement authentication (SCRAM, PLAIN, GSSAPI)
• Set up ACLs: Fine-grained authorization control
• Network Segmentation: Isolate Kafka cluster
• Audit Logging: Enable and monitor access logs
• Regular Updates: Keep Kafka version current
• Secret Management: Use secure credential storage
• Inter-broker Security: Secure broker-to-broker communication

Production Deployment Best Practices

# JVM Configuration
export KAFKA_HEAP_OPTS="-Xmx6g -Xms6g"
export KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 
-XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent 
-Djava.awt.headless=true"

# Server Properties
num.network.threads=8
num.io.threads=16
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
num.partitions=3
default.replication.factor=3
min.insync.replicas=2
unclean.leader.election.enable=false
log.retention.hours=168
log.segment.bytes=1073741824

Capacity Planning Guidelines

Troubleshooting Decision Tree

High Consumer Lag?
├── Yes → Check consumer processing time
│   ├── High → Optimize processing or scale consumers
│   └── Normal → Check partition count vs consumer count
└── No → Check other metrics

Frequent Rebalancing?
├── Yes → Check session.timeout.ms vs processing time
│   ├── Timeout too low → Increase session.timeout.ms
│   └── Processing too slow → Optimize or parallelize processing
└── No → Normal operation

Producer Errors?
├── Network errors → Check connectivity and broker health
├── Serialization errors → Validate message format
├── Authorization errors → Check ACLs and credentials
└── Timeout errors → Check broker load and configuration

Testing Strategy

Migration and Upgrade Strategies

Key Interview Topics Summary

1. Core Concepts: Topics, partitions, brokers, producers, consumers, offset management
2. Delivery Guarantees: At-most-once, at-least-once, exactly-once semantics
3. Consumer Groups: Load balancing, fault tolerance, rebalancing mechanisms
4. Performance: Batching, compression, partitioning strategies, tuning parameters
5. Monitoring: Key metrics, alerting strategies, troubleshooting approaches
6. Architecture: Event-driven patterns, CQRS, event sourcing, microservices integration
7. Operations: Deployment, scaling, security, disaster recovery
8. Ecosystem: Kafka Streams, Connect, Schema Registry integration