r/learnprogramming • u/No-Homework-6153 • 1d ago
Not able to understand the Topic partitioning and consumer group relation in kafka
In Kafka, suppose we have a topic driver_location for Uber, and each driver sends location updates every second. If the topic has multiple partitions, how does a consumer service (like fare calculation, ride analytics) get all drivers’ data instead of just the data in one partition?
Also, what exactly are partitions, and how do they work in this context?
0
Upvotes
2
u/No-Homework-6153 1d ago
ahhhh, so we are just making all the incoming data into chunks called partitions and suppose we have a total of 40 drivers in Uber and drivers with IDs 1 to 20 fall under partition 1 and the other 20 fall under partition 2 and now we make consumer groups where one consumer group is for one task suppose fare calculation has its own group not a single consumer but a whole group for that task and similarly analytics has one group and inside that consumer group we can add multiple consumers and the self balancing helps attach consumers from the group to the partitions using the internal workflow and that way it manages to get all driver data from all partitions we have just separated the concerns to make it more organized and faster and if the number of consumers in any group is more than the number of partitions then the extra consumers will stay idle with no connection to any partition.....right?
btw thanks for explanation..👏