Global Video Upload Service

Problem Statement

The client requested a video upload sevice capable of ingesting content from a globally distributed user base. The key requirements were scalability, security and reliability.

There were a number of challenges involved in this design. Firstly, we needed to make sure that malicious actors weren't able to sabotage the system by posting a large number of huge files. Secondly, as company was in the travel sector and each video was associated with a specific location, we had to minimise the user effort by auto-detecting the video location as much as was reasonably possible. Lastly, we had to integrate this service with a legacy monolith backend that powered some of the company's main B2C journeys.

Solution

The final solution uses the Serverless Framework to deploy a suite of AWS cloud infrastructure - mostly Lambda functions - to meet the business requirements. Most of the service's functionality was contained inside an AWS Step Functions workflow.

The security requirements are met by using presigned URLs in combination with checksum comparisons and rate limiting.

In order to tag videos to a particular location, FFmpeg and FFprobe were used to extract location metadata from raw video files which was then used to query the Google Places Reverse Geocoding API for a list of relevant locations that were sent to the user for review at the point of upload.

Integration with the legacy backend monolith was achieved through an orchestration and mapping layer that reshaped the input data into a format understood by the legacy system.

Lastly, the service was able to meet the scalability requirements through the reuse of Lambda execution context for caching of Secrets Manager secrets, the choice of Amazon S3 for blob storage, as well as of an Amazon SQS to decouple service components.

Architecture Diagrams

Full Service

State Machine Workflow

Successful State Machine Execution