Austin's SWE Notes 😭

    Kafka Redundancy

    To ensure redundancy, we can setup topic replication.

    # Creating a topic with a replication factor of 3
kafka-topics.sh --create --topic my-topic --partitions 10 --replication-factor 3 --bootstrap-server localhost:9092

    This means data does not only writen down to one Kafka Brokers but to multiple

    Pasted image 20240614184840.png

    For example in here, data that's written in Broker101 will then get replicated in Broker102. In the case that Broker102 failed, data is still available in Broker101 and Broker103

    A Kafka Broker Leader will have the main responsibility to receive the message and then pass it to Kafka Broker Replica

    Configure to guarantee replica

    We can configure ISR (In-sync replicas) so that we can have the minimum number of brokers needs to acknowledge that data is replicated, otherwise considered as failed.

    # Broker configuration in server.properties
min.insync.replicas=2
    ← Kafka Partition
    Kafka Stream →