Lambda Event Source Mapping

• Lambda polling from the service and invoke synchronously
• For this, Lambda has a Lambda Execution Role (IAM Role) to pull from the resource
• Lambda is polling from the following services (specify in the trigger box):
  • Kinesis Data Streams
  • SQS & SQS FIFO queue
  • DynamoDB Streams

DynamoDB Streams and Kinesis Data Stream

• Event source create iterator for each shard, and process items in order.
  • It starts with new items from beginning or from timestamp
• Processed items are not removed from the streams for other consumers to read them
• 1 Lambda invocation per stream shard or up to 10 batches processed per shard simultaenously
  • Can process multiple batches in parallel
  • in-order processing (the same order) is guaranteed for each partition key
• Error handling: if a function returns an error, the entire batch is reprocessed until the function succeeds, or the items in the batch expires. In that case, you can:
  • discard old events (go to a specific Destination)
  • restrict the number of retries
  • split the batch on error (workaround lambda timeout issues)

SQS and SQS FIFO

• Event source will poll from SQS (Long polling)
• Specify the batch size (1-10 messages)
• Recommended: Set queue visibility timeout to 6x the timeout of your lambda function
• To use a Dead Letter Queue, we need to setup on the SQS queue itself. Not on lambda since it's only for async
  • Or we can use Lambda Destinations for failure
• For in-order processing, we need to use SQS FIFO. Messages with the same GroupID will be processed in order.
  • The scaling == number of active message group
• SQS Standard Queue does not guarantee in-order processing
  • Lambda scales up automatically to process the queue as fast as possible
    • 60 more instances per minute to scale up
    • Up to 1000 batches of messages processed simuiltaneously
• Error handling: When error happens, batches are returned to the queue for other consumer to process.
  • Ocasionally event source mapping might receive same item twice even if no error occured
• Lambda delete items from the queue after they're processed successfully