Kafka Scaling

Architect

In here we dont need a load balancer, since Kafka Brokers are smart brokers that works when we just joining nodes

Producing message

Producer directly push message to the partition based on its in-memory metadata map. This is to save CPU and resources for kafka broker.

The metadata here has ages configured by metadata.max.age.ms. After this time, we will initialise the request for metadata again.

Also when we receive messages like LEADER_NOT_AVAILABLE or NOT_LEADER_OIR_NOT_FLOWER, this process of refreshing metadata triggered as well

sequenceDiagram
    participant P as Producer Client
    participant B as Broker (Metadata Provider)

    Note over P: Cache: 3 Partitions (0, 1, 2)
    Note over P: Timer hits 5 mins OR <br/>App notices mismatch
    
    P->>B: "Any changes to Topic X?"
    B-->>P: "Yes! Topic X now has 6 partitions."
    
    Note over P: Cache Updated: (0, 1, 2, 3, 4, 5)
    Note over P: Hash(Key) % 6 now results in 3
    
    P->>B: Sends message to Partition 3
    B-->>P: Success!

Consuming message