Siddhant Khare
Posted on • Updated on • Originally published at builtin.com
#aws #apigateway #s3 #programming
tl;dr; The AWS API Gateway has a payload size limit of 10MB, which was determined based on customer feedback and load testing during the service launch[1]. Currently, it is not possible to configure this limit, but AWS may consider enhancing this feature in the future for customers who require further limitations[1]. If a compressed payload smaller than 10MB is passed to the API Gateway, it will be accepted. However, if the uncompressed payload exceeds 10MB, the API Gateway will auto-decompress it and may reject the request with an "Error 413 Request entity too large"[1]. To work around this limit, one recommendation is to use Amazon S3 for large file transfers by returning an S3 Pre-signed URL in the API Gateway response, allowing direct uploads to S3[2]. This approach helps bypass the 10MB limit of the API Gateway and enables uploads directly to S3, which can handle larger file sizes up to 5TB with multipart uploads[3].
Introduction
In this blog post, we'll explore how to upload large files to AWS S3 using presigned URLs. Originally, our file upload mechanism was handled through AWS API Gateway, which is limited to accepting files up to 10 MB. To accommodate larger files, we've implemented a solution using S3 presigned URLs, allowing files to be uploaded directly from the client interface without size limitations imposed by API Gateway.
Process Overview
The process involves two main steps:
- Fetch the presigned URL from S3 via an API Gateway → Lambda setup.
- Use the obtained URL to upload the file directly from the client.
Diagram explanation
Lambda Function for Generating Presigned URLs
First, let's look at the Lambda function, which is responsible for generating the presigned URL. We'll use Node.js for the Lambda function:
const { S3Client, PutObjectCommand } = require('@aws-sdk/client-s3');const { getSignedUrl } = require('@aws-sdk/s3-request-presigner');const client = new S3Client({ region: 'eu-west-2', signatureVersion: 'v4'});exports.handler = async (event, context) => { // Define the parameters for the S3 bucket and the object key const params = { Bucket: 'YOUR_BUCKET_NAME', Key: 'OBJECT_KEY' } const expires = 3600; // Set expiration time for the signed URL (in seconds) try { const url = await getSignedUrl(client, new PutObjectCommand(params), { expiresIn: expires }); return { headers: { 'Access-Control-Allow-Origin': '*' }, body: JSON.stringify({ url }) } } catch (error) { return { headers: { 'Access-Control-Allow-Origin': '*' }, body: JSON.stringify({ error: error.message }) } }};
Client-Side Upload Using React
Next, let's discuss how to use this presigned URL on the client side to upload a file. Here we use React:
import axios from 'axios';// Function to upload the file using the presigned URLexport async function upload(file) { const getUrlResponse = await getSignedUrl(); const url = getUrlResponse.body; // Handle cases where the URL is not obtained if (!url) return; await uploadFile(url, file);}// Function to retrieve the signed URLexport async function getSignedUrl() { const result = await axios.get('API_GATEWAY_URL') .then(response => { return { body: response.data.url } }) .catch(err => { return { body: null } }); return result;}// Function to perform the file uploadexport async function uploadFile(url, file) { const result = await axios.put(url, file, { headers: { 'Content-Type': file.type } }).then(response => { return { body: response.data } }).catch(err => { return { body: null } }); return result;}
CORS Configuration on S3
One common issue during implementation is CORS errors. Here's how you can configure CORS in your S3 bucket settings:
{ "AllowedHeaders": ["*"], "AllowedMethods": ["PUT"], // Ensure 'PUT' is allowed for file uploads "AllowedOrigins": ["http://localhost:3000"] // Adjust this according to your client's URL}
Conclusion
By using AWS S3 presigned URLs, we can bypass the limitations of AWS API Gateway for large file uploads, offering a scalable and secure solution directly from client applications. This method not only simplifies the process but also keeps your data secure during transit.
Stay Connected and Get More Insights
If you found this guide helpful and are dealing with similar challenges, don't hesitate to reach out for personalized consulting at Superpeer. For more tech insights and updates, consider following me on GitHub. Let's innovate together!
Top comments (3)
Subscribe
Constantin
Constantin
Web-Application developer, Frontend/Backend on AWS connecting to big corporate legacy systems ;)
-
Location
Berlin, Germany
-
Joined
• May 13
- Copy link
Small caviat, bucket names can be abused for a financial ddos, since you are also paying for authorized api calls against your bucket.
repost.aws/questions/QUbqrBw4QhQIu...
In general using presigned URLs is a good idea, however one should be aware of the potential attack vector
Frank the tank
Frank the tank
Tries to write code here and there.
-
Joined
• May 14
- Copy link
Use multipart uploads and problem solved.
amythical
amythical
90s Dot com survivor - built a messenger and ad server in 90sSuccessful Exit from a Digital Media agencyH/W IOT startup built Social Toys https://shorturl.at/amQU2Now building chopchop.video
-
Email
-
Location
Mumbai, India
-
Education
Mumbai University
-
Work
Co-Founder chopchop.video
-
Joined
• May 18
- Copy link
But the payload would still go to a server first and then pushed to a storage like S3. Wont that mean running a large server instead of a smaller instance instead and moving the upload functionality to the client and S3.
Thoughts?
For further actions, you may consider blocking this person and/or reporting abuse