Partitions

This is the introduction to Apache Kafka on partitions. Now, you probably know Kafka is a distributed system, so it's designed to operate across a number of computers and look on the outside world like it's just one big computer, right? Well, given that, if a topic were constrained to live entirely on one machine, just one node in that cluster, it would be a bit of a bummer. It would put a radical limit on the ability of Kafka to scale because then a topic could never get any bigger than the biggest node you could deploy and the storage on that node, and you know, that's not so great. So partitioning is how Kafka handles that.

Partitioning takes that single topic log that we talked about in the last module and breaks it up into multiple logs. This topic is partitioned, split into pieces. This is gonna have some effects on the concept of message ordering. We'll see how this works out, but in short, we don't have global ordering, at least not a strict guarantee of global order within a partitioned topic. We only have that within each partition. I'll show you an animation of that in just a little bit. Of course, you're not limited to just three partitions in a topic. You can have hundreds, you can have thousands. If you want, topics can get very, very large. Kafka in the current version, open source, Apache Kafka, supports round numbers, about 2 million partitions in releases. That's around Kafka 4.0 at the time of this recording. That's how scalable Kafka can be.

Of course, once we break a topic up into these partitions, now we need a way of deciding which messages get written to which partition, right? You've got somebody writing messages to this topic, and now that topic is split into pieces, I need to know which partition a given message goes to. Now, if a message has no key, and remember, we think of messages as key-value pairs in the simplest form, that key is null, it's empty, it's just not there, then the messages you write will be distributed round-robin among the partitions. Each message you write just gets assigned to the next partition, load is distributed evenly, and it's a wonderful world.

Well, mostly wonderful. Notice those messages from thermostat 42. You see, they all went to different partitions. They're not in order. They're kind of in order. Roughly speaking, stuff earlier on in the partition is gonna be older than stuff later on in the partition, but I don't have a guarantee of order. When I read things out of this partition, I don't know that I'm gonna get those in the same order they went into it.

If you want ordering by something like sensor ID, in this case, there's a way to do that. If the sensor ID is the message key, then what happens is that key gets run through a hash function, mod the number of partitions, and that just tells us the partition number to write to. So in this message for that key, that's partition two. So that message gets written to partition two. Likewise, the next one, 42, gets written to the same partition. You see, we have strict ordering for any messages of the same partition key. Now, different keys, anything goes, right? They could be assigned to whatever partition happens to win the hash function lottery, but we get an ordering guarantee on messages of a common key, and that can be very, very useful in some use cases.

To sum it up, Kafka partitions are what make Kafka scalable, efficient, and reliable. By dividing topics into partitions, Kafka can distribute workloads across a cluster and handle massive amounts of data, massive amounts of IO, pub-sub activity happening, and preserve message order when you need it. Understanding how partitions work, especially how they're assigned and balanced, helps you get the most out of Kafka.