Austin's SWE Notes 🧠

Ticketmaster - Ticket Booking System

Functional requirements

  1. User should be able to view event
  2. User should be able to search for events
  3. User should be able to book tickets to events

Out of scope: (Check with interviewer if they want to move any of this to the functional requirements)

  1. User should be able to view their booked events
  2. Admins or event coordinator should be able to add event
  3. Popular events should have dynamic pricing

Non functional requirements

  1. System should prioritise availability for searching & viewing events, but consisteny for booking events
  2. System should be scalable and handle high throughput for popular event (max 10 million user 1 event)
  3. Search latency should be low (< 500ms)
  4. System is read heavy, need to be able to support high read throughput (100:1)

Out of scope: (nice to have)

  1. System should protect user data and adhere GDPR (data encryption, anonymize)
  2. System should be fault tolerant
  3. System should provide secure transactions for purchases
  4. System should be well tested and easy to deploy
  5. System should be regular backups

Core entities

  1. Event: store the information about events i.e date, description, type
  2. User: represents individual iiteracting with the system
  3. Performer: artist name, company etc
  4. Venue: venue name, address, seat map, seat layout
  5. Ticket: contains the information about the events i.e event ID, seat details, pricing
  6. Booking: User ticket purchase detail i.e user id, ticket ids

[!note] Booking can also be combined with ticket
We can also combine booking with ticket itself, however in the case where 1 user booking contains multiple tickets, it could be useful

API

1. User to view event

GET /v1/events/:eventId -> Event & Venue & Performer & Ticket[]


2. User can search for event

GET /v1/events/search?keyword={keyword}&start={start_date}&end={end_date}&pageSize={page_spize}&page={page_number} -> Event[]


3. User can purchase the ticket for the event

POST /v1/events/:eventId/booking -> Booking[]
{
    ticketIds: string[]
    paymentDetails: ...
}

[!note]
In production system, normally we dont send the paymentDetails as the post request, instead the user will send payment data to 3rd party payment system i.e Stripe, which we will use the transactionId provided by the user to validate against Stripe API

High level design

1) User should be able to view the events

User will make a GET /v1/events/:eventId to API gateway which route the request to Event Service. The Event Service will then query our database for information and display back to the user

2) User should be able to search for events

In here, we can create a search service to handle searching. We technically can put in the same service as well but we want to CQRS (Command Query Responsibility Segregation) to scale easily — hence the split of search service and event service.

User can make a GET /v1/events/search?keyword={key_word}... to our API Gateway, the API gateway will route this to our search service. If necessary, we can have a layer of load balancer in the front to scale out the search service as well.

Search service then query the database and return us back the event information for the client.

3) Users should be able to book tickets to event

In this case, we need to handle concurrent booking. This is a good time we choose our database technology

Anything from MySQL to DynamoDB is fine as long as it support ACID (DynamoDB now support ACID as well even thought it's often used as BASE style)

In order to guard the seats for booking, we need to use either row-level locking or Optimistic Locking of some sort

This time, we introduce these in our database

  1. Bookings: store the detail for each booking, 1 booking can contains multiple ticket to handle the scenario where 1 person book multiple seats this will include
    1. userId: the user who authored this booking
    2. ticketIds: list of tickets id for this booking
  2. Tickets: store the detail of each ticket
    1. eventId
    2. seat: the detail info of a seat
    3. pricing
    4. status: AVAILABLE | BOOKED

Booking Service: microservice responsible for the core functionality of ticket booking process, interact with the databases that store data on bookings and tickets

  1. Interact with stripe to process the payment
  2. Once payment received, communicate with Bookings and Tickets to update the payment

Payment processor (stripe): external service responsible for handling payment transactions, once the payment processed it will notify the booking service

[!note]
You can see in here that multiple microservices share the same database. This is fine, there is a belief that only 1 service per database but in production, many company share 1 database across services. In our case, we need ACID for booking and splitting the database only make it more complicated

Potential deep dives

1) How do we improve the booking expereience by reserving tickets

When no reserving tickets mechanism, the user need to try to buy to see if they successfully reserve or not, that's very bad UX.

We want to implement a count down timer for ticket reservation as often seen in ticket booking site

Bad solution, Pessimistic locking with For Update

See Pessimistic locking.

One way to do this is to use SELECT FOR UPDATE which will pre-lock the rows. During this time, other transactions touch rows from SELECT FOR UPDATE will blocked until the lock is released.

However, doing this will keep the transaction, database connection oepn for a long period. PostgreSQL support lock_timeout to fail transaction that wait too long for locks but this is not graceful solution.

This approach also not scale under high load, prolonged row lock could create wait time for other users, in terms of application crash or network issues, it could leave the lock in uncertain stage.

Great solution, Lock using database column

We have for each row, we set the status and also the expiration.

  1. We begin a transaction
  2. We check to see if the current ticket is either AVAILABLE or (RESERVED but expired)
  3. We update the ticket to RESERVED with expiration = now + 10 mins
  4. We commit the transaction

Challenges: our read operation are going to be slightly slower by needing to filter by status and expiration.

In order to solve this, we can create a Compound key index with (event_id, status, expiration) (see Database index (Secondary Key)) the database will auto update the index for us to search on the latest result.

Another way is to create a MATERIALIZED view which contains a snapshot of the available tickets and call REFRESH MATERIALZIED VIEW .... However, this will create stale result

Great solution, Lock using distributed lock with TTL

As you can see from our previous solution, we need to use query mechanism to check if something is expire or not, this would not work well in high traffic.

When we use Redis, it natively support row-level TTL which we can leverage for key expiration. Lock acquisition and release are extremely fast under high concurrency

  1. User select a seats, it acquire the lock on redis with predefined TTL.
  2. If the user complete the purcahse, database update to BOOKED and the lock manually release by the application
  3. If TTL expire, the seat become available for another user

From Redis side, we can use SET key value NX EX seconds for the expiry, the key could be lock:event:{eventId}:{seatId} and the value could be the userId so we know who owns the lock

For multi seats, you can acquire the locks sequentially per ticket, if lock fail, we release the one we already acquired.

Challenges

  1. How do we show unavailable seat on the seat map now the reservation responsibility has been handed to redis?
    • We need a way to track which seats are reserved and store them on redis. To do this, we can use a Sorted Set which the key is the expireAt (note that sorted set the lowest value will be first)
    • When adding a seat to reserved, we use ZADD event:{eventId}:reserved <expiredAt> ticketId
    • When we query, we do 2 things
      • ZREMRANGEBYSCORE event:{eventId}:reserved -inf now: remove all the expired reservation
      • ZRANGEBYSCORE event:{eventId}:reserved (now +inf: display seat reservation that the expiry is > now
  2. If the distributed lock goes down, how are we gonna handle failure.
  3. What if TTL expire during payment.
    • We use transactional, Optimistic Locking to make sure that only 1 person can book still to avoid double book. This make sure only 1 person can successfully book, the other we just issue a refund
  1. User select a seat from the seat map, trigger a POST /v1/events/:eventId/bookings
  2. Request will forward to API gateway into booking service
  3. Redis will lock that ticket for TTL 10 minutes
  4. Booking Service write a new Booking entry with status in progress
  5. Service return bookingId to the user, user will be routed to payment page to finish the payment
    1. Lock for this page is corresponding to Redis TTL (10 minutes). If this lock expired, the ticket is available for other user to purchase
  6. User fills out payment details on checkout page. Client use Stripe.js to tokenize the card details — our server never see card numbers. Client send the resulting payment token and bookingId to our server.
    • Stripe will then call our system via webhook that the payment was successful (we need to expose a webhook endpoint on our service)
  7. Once payment is successful, system webhook retrieve bookingId embeded in Stripe metadata.
    1. BookingService initiate a database transaction to concurrently update the Ticket and Booking tables.
      1. The status of the ticket in the booking is change to sold in the Ticket table.
      2. The Booking status now change to CONFIRMED
    2. Webhook should be Idempotent so that Stripe can retry on failure without duplicating the state change.
      1. We can use WHERE booking_id=:booking_id AND status='IN PROGRESS' to make sure that even after retry, if status is CONFIRMED this won't have any DB modification
  8. Ticket is sold

We need to scale the Event Service out using

  1. Load balancing / auto scale
  2. Redis for caching popular content
    • Prioritise cache for high read data such as event details, names, dates, venue information
    • Consider some cache invalidation strategy such as Write through > Write through with invalidation
    • We can adjust TTL for different fields i.e TTL can be long for static data like event venue, but short for event availability

3) How will the system ensure good user experience during high-demand with millions simultaneously booking tickets

With popular event, seat map will stale very quickly. Users will be frustrated if they repeatedly click on a seat to find out if it's already booked.

We need a way to ensure that the seat map is always up to date, users are notified in real-time

Good solution: Server-Sent Events (SSE)

We use SSE to update the client in real time as soon as seat map is booked or reserved without the user refreshing the page

Challenges: for extremely popular event, it creates a lot of load on the server, the heat map will immediately fill up.

For extremely popular event, we need a virtual watiing queue to avoid the seat map to immediately fill up. The queue can sit in front of booking service.

  1. When user access the booking place, they're placed into this virtual queue
  2. We at the same time establish an SSE with the client and add to the queue.
    • The queue itself we can use Redis Sorted Set with timestamp. SSE is fired if the user move out from the wait queue to go to the booking page
  3. We perdiotically deque users from the front of the queue. Notify them via SSE
    • For the user that we want to dequeu, we add them in the admitted:{eventId} set with a TTL. This allows us to reject users who manually go to the booking page itself without the app letting them in

4) How can you improve search to ensure we meet our low latency requirements?

Currently we are doing SQL query which is very slow

-- slow query
SELECT * 
FROM Events
WHERE name LIKE '%Taylor%' 
  OR description LIKE '%Taylor%'

We can create index on different table such as Event, Performer and Venue to improve the query performance. We can think of a way to optimise the SQL query however it is normally not good enough.

Great solution: Full-text indexes in the db

postgres has built-in full-text search using tsvector and GIN indexes. These make queries for speicific string like "Taylor" or "Swift" much faster than doing a full table scan

Challenges:

  1. This means we need to provide additional storage space, can be slower to query than standard index
  2. Difficult to maintain as they require special handling in both queries and database

Great solution: Use a full-text search engine like ElasticSearch

Elasticsearch operate using Inverted Index which is highly efficient for search operation.

We can use Change Data Capture (CDC) for near-real time or real-time data synchronisation from postgres to elasticsearch.

  • This captures changes from database: updates,inserts, deletes and replicate them to Elasticsearch index
  • This works by using WAL (Write A-head Logging)

We can enable fuzzy search functionality with elasticsearch which allow error tolerance as well

Challenges

  1. Adding elasticsearch cluster adds additional infrastructure complexity and cost
  2. Keeping elasticsearch index synchronised with postgres can be complex to maintain data consistency

5) How can you speed up frequently repeated search queries and reduce load on search infras

While we can use redis to store the result of frequently executed search query:

{
  "key": "search:keyword=Taylor Swift&start=2021-01-01&end=2021-12-31",
  "value": [event1, event2, event3],
  "ttl": 60 * 60 * 24 // 24 hours
}

Cache invalidation can be challenging, frequent cache misses can lead to increase load on search infrastructure, especially during peak times

Great solution: Implement query result caching and edge caching techniques

ElasticSearch has built-in caching capabilities that we can leverage to store frequent query. This will reduce the query processing on the search engine.

  • ElasticSearch maintains query caches at shard level for filter results, plus separate shard-level request cache for caching full search response

We can also use CDN to cache the search result geographically closer to user, reduce the latency and improve response time.

  • NOTE: This only makes sense when the search is not personalised and common

Challenges

  1. Consistency between cached data and real time data requires sophiscated synchronisation mechanism. You need to invalidate the cache whenever underlying data changes
  2. Demands more infrastructure support including CDN and managing adaptive caching system

Final design

← High-Level-Design-3.Excalidraw
Deep-Dive-Design-1.Excalidraw →