Kinesis Producers

• Put data into Kinesis Data Streams
• Data record consist of
  • Sequence number (unique per partition key within shard)
  • Partition key (must specify while put records into stream)
    • Kinesis will hash partition key to send data to relevant shard
    • Make sure you distribute the partition key equally to avoid everything goes to the same partition. See also: Kinesis Operation
  • Data blob (up to 1MB)
• Producers:
  • AWS SDK: simple producer
  • Kinesis Producer Library (KPL), C++, Java, Batch, Compression, Retries
• Use PutRecord API
• Write throughput: 1MB/sec or 1000 records/sec per shard
  • Use batching with PutRecord to reduce costs & increase throughput

ProvisionedThroughPutException

• Happens when we try to put more throughput than provisioned
• Solution:
  • Retries with Exponential Backoff (any AWS service)
  • Use highly distributed partition key
  • Increase shards (scaling)
    • Increasing 1 more shard == having 1 MB/sec more