KnowBite KnowBite

Data Engineering · · 1 min read

Best practices for scaling large consumer groups on Amazon MSK

As consumer groups on Amazon MSK scale to thousands of members, the metadata record Kafka writes during rebalances can exceed the 1 MB limit and stall the group. Learn how to estimate metadata size, raise the topic-level limit safely, plan capacity, and apply complementary strategies for scaling large consumer groups.

Read original on AWS Big Data Blog