Write Scale

Write scale

Horizontal scale

Redis Cluster

• Redis cluster does not support consistent hashing, it's CRC16 with 16384 fixed hash slots
• Redis cluster handle the slot number which server hold it

We need to select a good partitioning key, normally userId is sufficient to make it evenly distributed. You can also do a composite parition key.

• We want to select a key that spread out evenly
• However when sharding we need to check how the read pattern is, it's bad if we need to go and read for every shard

Queue

Good for burst absorption queue act as buffer, smoothing out traffic spikes.
NOTE Using queue, we need to make sure that the database can handle the write as well.

Load shedding

Load shed drop the non-important write and write the important one

For example, if the user keep posting their coordination for every single seconds and we have excess number of user. We can choose to drop some write since requests keep in coming

Batching

Batch the write together and send to the database. This can be done in the application layer or have a middle microservice to handle the batching

Hierarchical Aggregation

Like comment update in real time, instead of the app distribute to all user, we put user into rooms and broadcast

Root processor can scale into write processor as well

Common deep dive

"How do you handle resharding when you need to add more shards?"

If we start with 8 shard, we now need 16 how do we reshard.

• Bad approach: take the thing offline, rehash all data, move to new shard => Not good
• Good approach: Good migration, new data write targets to both locations (old shard and new shard) while async migrate the data over to new shard. Read preferred new shard before fall back to old shard

"What happens when you have a hot key that's too popular for even a single shard?"

Duplicate the key into multiple shard

• Separate the same key into key1-shard-0, key-1-shard-1, key-1-shard-2 instead of having only 1 shard.
• But this means we increase our data volumn by k time, read volumn by k

Split the key dynamically

• If a hot key come in, agree that we will use the range from a to b to split this key
  • key1-shard-a...key1-shard-b
• However it's hard since both reader and writer need to agree on the range and have it synchronised.
• Most production system use first approach