Kafka Partition

Is a storage unit that holds Kafka Events (Message, Record). Belongs to a Kafka topic

The partition is basically a list of Kafka Events (Message, Record) — in which the indices are called Kafka Offset.

Use case

Scalability

Kafka partition can help with scalability. Which multiple partition can be distributed across multiple Kafka Brokers which helps to scale horizontally

It's multiple Kafka Consumer can now read the kafka topic in parallel from multiple brokers.

Redundancy

Kafka keeps more than one copy of the same partition across multiple brokers for replicas purpose.

Writing into partition

We can use a partition key to specify which partition the record will be write to.

Without a partition key, by default, kafka will use round-robin assignment — which means distributing the Kafka Events (Message, Record) across all partitions.

Reading from partition

Kafka Consumer will use Kafka Offset as a pointer to consume the message. It acts like a cursor for the consumer.

After reading a message, consumer advance to the next offset.