diff --git a/README.md b/README.md index cf8c338..7b93fdd 100644 --- a/README.md +++ b/README.md @@ -1,288 +1,171 @@ -# Serverless image resizer +# Serverless Image Resizer -[![LocalStack Pods Launchpad](https://localstack.cloud/gh/launch-pod-badge.svg)](https://app.localstack.cloud/launchpad?url=https://github.com/localstack/sample-serverless-image-resizer-s3-lambda/releases/download/latest/release-pod.zip) [![GitHub Actions](https://github.com/localstack-samples/sample-serverless-image-resizer-s3-lambda/actions/workflows/integration-test.yml/badge.svg)](https://github.com/localstack-samples/sample-serverless-image-resizer-s3-lambda/actions/workflows/integration-test.yml) [![CircleCI](https://dl.circleci.com/status-badge/img/gh/localstack-samples/sample-serverless-image-resizer-s3-lambda/tree/main.svg?style=shield)](https://dl.circleci.com/status-badge/redirect/gh/localstack-samples/sample-serverless-image-resizer-s3-lambda/tree/main) [![AWS CodeBuild](https://codebuild.eu-central-1.amazonaws.com/badges?uuid=eyJlbmNyeXB0ZWREYXRhIjoiUUVDZExZT0ovUm5YejlKcHlXeGpuT1pRaC9hSzdDcFJVYlMvZzl0emtnWW5qcVdQUGY3YlJYWnBJeEdoOHd0OVkvd29XTUxuL2p3d3hVS0NFRnFlTzhRPSIsIml2UGFyYW1ldGVyU3BlYyI6IjQrdHhqVGY1N24ycTEvQkMiLCJtYXRlcmlhbFNldFNlcmlhbCI6MX0%3D&branch=codebuild-sample)](https://eu-central-1.codebuild.aws.amazon.com/project/eyJlbmNyeXB0ZWREYXRhIjoiOGpTQnR0a0J6ZnYwN3hNQ21DVkFoUU8zWTc4TExSaGk0b2p5UkVyNWhHSXhLSWZUSWt3eE1PUnpLZTRMWld2U3l3bVBWa2Frc084YjJ6UFZDRjNlcTc0U0xOa2lqVU1qZXdJMUFzdEVudz09IiwiaXZQYXJhbWV0ZXJTcGVjIjoib1FaZmhKMHZkc0NTbmdqcSIsIm1hdGVyaWFsU2V0U2VyaWFsIjoxfQ%3D%3D) [![GitLabCI](https://gitlab.com/localstack.cloud/samples/sample-serverless-image-resizer-s3-lambda/badges/main/pipeline.svg?ignore_skipped=true)](https://gitlab.com/localstack.cloud/samples/sample-serverless-image-resizer-s3-lambda) -| Key | Value | -|--------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| Environment | | -| Services | S3, SSM, Lambda, SNS, SES | -| Integrations | AWS SDK, AWS CLI, GitHub actions, pytest | -| Categories | Serverless, S3 notifications, S3 website, Lambda function URLs, LocalStack developer endpoints, JavaScript, Python | | -| Level | Intermediate | +| Key | Value | +| ------------ | ------------------------------------------------------------------------------------------ | +| Environment | LocalStack, AWS | +| Services | S3, Lambda, SNS, SES, SSM | +| Integrations | AWS SDK, AWS CLI, pytest | +| Categories | Serverless, S3 notifications, S3 website, Lambda function URLs | +| Level | Intermediate | +| Use Case | Lambda DevX, Integration Testing | +| GitHub | [Repository link](https://github.com/localstack/sample-serverless-image-resizer-s3-lambda) | ## Introduction -This is an app to resize images uploaded to S3 in a serverless way. -A simple web fronted using HTML and JavaScript provides a way for users to upload images that are resized and listed. -We use a Lambda to generate S3 pre-signed URLs so the upload form can upload directly to S3 rather than going through the Lambda. -S3 bucket notifications are used to trigger a Python Lambda that runs image resizing. -Another Lambda is used to list all uploaded and resized images, and provide pre-signed URLs for the browser to display them. -We also demonstrate how Lambda failures can submit to SNS, which can then trigger an SES email. -Using the LocalStack internal `/_localstack/aws/ses` endpoint, we can run end-to-end integration tests to verify that emails have been sent correctly. - -Here's a short summary of AWS service features we use: -* S3 bucket notifications to trigger a Lambda -* S3 pre-signed POST -* S3 website -* Lambda function URLs -* Lambda SNS on failure destination -* SNS to SES Subscriptions -* SES LocalStack testing endpoint - -Here's the web application in action: +This sample demonstrates how to build a serverless image resizing application using AWS Lambda, S3, and related services. The application features a simple web frontend that allows users to upload images, which are then automatically resized using Lambda functions triggered by S3 events. To test this application sample, we will demonstrate how you use LocalStack to deploy the infrastructure on your developer machine and test the complete workflow locally. We will also show how to use Lambda hot reloading for rapid development cycles and comprehensive integration testing to ensure the application works end-to-end. -https://user-images.githubusercontent.com/3996682/229314248-86122e9e-0150-4292-b889-401e6fb8f398.mp4 +## Architecture -Moreover, the repo includes a GitHub actions workflow to demonstrate how to run end-to-end tests of your AWS apps using LocalStack in CI. -The GitHub workflow runs a set of integration tests using pytest. +The following diagram shows the architecture that this sample application builds and deploys: -## Architecture overview +![Architecture overview](https://user-images.githubusercontent.com/3996682/229322761-92f52eec-5bfb-412a-a3cb-8af4ee1fed24.png) -![Screenshot at 2023-04-02 01-32-56](https://user-images.githubusercontent.com/3996682/229322761-92f52eec-5bfb-412a-a3cb-8af4ee1fed24.png) +- [S3 Buckets](https://docs.localstack.cloud/aws/services/s3/) for storing original and resized images, plus hosting the static website +- [Lambda Functions](https://docs.localstack.cloud/aws/services/lambda/) for generating pre-signed URLs, image processing, and listing images. +- S3 Event Notifications to trigger the resize Lambda when new images are uploaded. +- [SNS](https://docs.localstack.cloud/aws/services/sns/) and [SES](https://docs.localstack.cloud/aws/services/ses/) for error notifications and email alerts. +- [SSM Parameter Store](https://docs.localstack.cloud/aws/services/ssm/) for configuration management. ## Prerequisites -### Dev environment - -Make sure you use the same version as the Python Lambdas to make Pillow work. -If you use pyenv, then first install and activate Python 3.11: - -```bash -pyenv install 3.11.6 -pyenv global 3.11.6 -``` - -```console -% python --version -Python 3.11.6 -``` - -Create a virtualenv and install all the development dependencies there: - -```bash -python -m venv .venv -source .venv/bin/activate -pip install -r requirements-dev.txt -``` - -## Instructions - -You can set up and deploy the sample application on LocalStack by executing the commands in our Makefile. First, create a `.env` file using the provided `.env.example` file as a template, and include your LocalStack token in it. Then, run `make start` to initiate LocalStack on your machine. +- [`localstack` CLI](https://docs.localstack.cloud/getting-started/installation/#localstack-cli) with a [`LOCALSTACK_AUTH_TOKEN`](https://docs.localstack.cloud/getting-started/auth-token/). +- [AWS CLI](https://docs.localstack.cloud/user-guide/integrations/aws-cli/) with the [`awslocal` wrapper](https://docs.localstack.cloud/user-guide/integrations/aws-cli/#localstack-aws-cli-awslocal). +- [Python 3.11](https://www.python.org/downloads/) (same version as the Lambda runtime) +- [`make`](https://www.gnu.org/software/make/) (**optional**, but recommended for running the sample application) -Next, execute `make install` to install needed dependencies. +## Installation -After that, launch `make terraform-setup` to provision the infrastructure on LocalStack using Terraform CLI and its scripts. Alternatively, run `make awslocal-setup` to set up the infrastructure using `awslocal`, a wrapper for the AWS CLI. +To run the sample application, you need to install the required dependencies. -If you prefer, you can also follow these step-by-step instructions for a manual deployment. - -### LocalStack - -Start LocalStack Pro with Auth Token: - -```bash -LOCALSTACK_AUTH_TOKEN=... localstack start (-d) -``` - -### Terraform - -To create the infrastructure using Terraform, run the following commands: +First, clone the repository: ```shell -cd deployment/terraform -tflocal init -tflocal apply --auto-approve -``` - -We are using the `tflocal` wrapper to configure the local service endpoints, and send the API requests to LocalStack, instead of AWS. - -### AWS CLI - -You can execute the following commands to set up the infrastructure using `awslocal`. All the commands are also available in the `deployment/awslocal/deploy.sh` script. - -#### Create the buckets - -The names are completely configurable via SSM: - -```bash -awslocal s3 mb s3://localstack-thumbnails-app-images -awslocal s3 mb s3://localstack-thumbnails-app-resized -``` - -#### Put the bucket names into the parameter store - -```bash -awslocal ssm put-parameter --name /localstack-thumbnail-app/buckets/images --type "String" --value "localstack-thumbnails-app-images" -awslocal ssm put-parameter --name /localstack-thumbnail-app/buckets/resized --type "String" --value "localstack-thumbnails-app-resized" +git clone https://github.com/localstack/sample-serverless-image-resizer-s3-lambda.git ``` -#### Create the DLQ Topic for failed lambda invokes +Then, navigate to the project directory: -```bash -awslocal sns create-topic --name failed-resize-topic +```shell +cd sample-serverless-image-resizer-s3-lambda ``` -Subscribe an email address to it (to alert us immediately if an image resize fails!). +Create a virtual environment and install the development dependencies: -```bash -awslocal sns subscribe \ - --topic-arn arn:aws:sns:us-east-1:000000000000:failed-resize-topic \ - --protocol email \ - --notification-endpoint my-email@example.com +```shell +python -m venv .venv +source .venv/bin/activate +pip install -r requirements-dev.txt ``` -#### Create the lambdas +## Deployment -##### S3 pre-signed POST URL generator +Start LocalStack with the `LOCALSTACK_AUTH_TOKEN` pre-configured: -This Lambda is responsible for generating pre-signed POST URLs to upload files to an S3 bucket. - -```bash -(cd lambdas/presign; rm -f lambda.zip; zip lambda.zip handler.py) -awslocal lambda create-function \ - --function-name presign \ - --runtime python3.11 \ - --timeout 10 \ - --zip-file fileb://lambdas/presign/lambda.zip \ - --handler handler.handler \ - --role arn:aws:iam::000000000000:role/lambda-role \ - --environment Variables="{STAGE=local}" +```shell +localstack auth set-token +localstack start ``` -Create the function URL: +To deploy the sample application, run the following command: -```bash -awslocal lambda create-function-url-config \ - --function-name presign \ - --auth-type NONE +```shell +bin/deploy.sh ``` -Copy the `FunctionUrl` from the response, you will need it later to make the app work. - -#### Image lister lambda - -The `list` Lambda is very similar: - -```bash -(cd lambdas/list; rm -f lambda.zip; zip lambda.zip handler.py) -awslocal lambda create-function \ - --function-name list \ - --handler handler.handler \ - --zip-file fileb://lambdas/list/lambda.zip \ - --runtime python3.11 \ - --role arn:aws:iam::000000000000:role/lambda-role \ - --environment Variables="{STAGE=local}" -``` +The output will show the Lambda function URLs that you can use in the web application: -Create the function URL: +```shell +Fetching function URL for 'presign' Lambda... +https://abcdef123456.lambda-url.us-east-1.localhost.localstack.cloud:4566/ +Fetching function URL for 'list' Lambda... +https://789012345678.lambda-url.us-east-1.localhost.localstack.cloud:4566/ -```bash -awslocal lambda create-function-url-config \ - --function-name list \ - --auth-type NONE +Now open the Web app under https://webapp.s3-website.localhost.localstack.cloud:4566/ +and paste the function URLs above (make sure to use https:// as protocol) ``` -#### Resizer Lambda - -```bash -( - cd lambdas/resize - rm -rf package lambda.zip - mkdir package - pip install -r requirements.txt -t package --platform manylinux2014_x86_64 --only-binary=:all: - zip lambda.zip handler.py - cd package - zip -r ../lambda.zip *; -) -awslocal lambda create-function \ - --function-name resize \ - --runtime python3.11 \ - --timeout 10 \ - --zip-file fileb://lambdas/resize/lambda.zip \ - --handler handler.handler \ - --dead-letter-config TargetArn=arn:aws:sns:us-east-1:000000000000:failed-resize-topic \ - --role arn:aws:iam::000000000000:role/lambda-role \ - --environment Variables="{STAGE=local}" -``` +## Testing -#### Connect the S3 bucket to the resizer lambda +Visit the web application at `https://webapp.s3-website.localhost.localstack.cloud:4566/` and paste the function URLs from the deployment output into the form fields. You can then upload an image and see it get automatically resized. -```bash -awslocal s3api put-bucket-notification-configuration \ - --bucket localstack-thumbnails-app-images \ - --notification-configuration "{\"LambdaFunctionConfigurations\": [{\"LambdaFunctionArn\": \"$(awslocal lambda get-function --function-name resize | jq -r .Configuration.FunctionArn)\", \"Events\": [\"s3:ObjectCreated:*\"]}]}" -``` +https://user-images.githubusercontent.com/3996682/229314248-86122e9e-0150-4292-b889-401e6fb8f398.mp4 -#### Create the static s3 webapp +You can run full end-to-end integration tests using the following command: -```bash -awslocal s3 mb s3://webapp -awslocal s3 sync --delete ./website s3://webapp -awslocal s3 website s3://webapp --index-document index.html +```shell +pytest tests/ ``` -#### Using the application - -Once deployed, visit http://webapp.s3-website.localhost.localstack.cloud:4566 +## Use Cases -Paste the Function URL of the presign Lambda you created earlier into the form field. -```bash -awslocal lambda list-function-url-configs --function-name presign -awslocal lambda list-function-url-configs --function-name list -``` - -After uploading a file, you can download the resized file from the `localstack-thumbnails-app-resized` bucket. +### Lambda DevX -### Testing failures +This sample demonstrates LocalStack's Lambda hot reloading capability, which enables immediate reflection of code changes without redeployment. With hot reloading, you can achieve fast feedback cycles during development. -If the `resize` Lambda fails, an SNS message is sent to a topic that an SES subscription listens to. -An email is then sent with the raw failure message. -In a real scenario you'd probably have another lambda sitting here, but it's just for demo purposes. -Since there's no real email server involved, you can use the [SES developer endpoint](https://docs.localstack.cloud/user-guide/aws/ses/) to list messages that were sent via SES: +To enable hot reloading for the `list` Lambda function, use the special `hot-reload` bucket name with an absolute path to your Lambda code: -```bash -curl -s http://localhost.localstack.cloud:4566/_aws/ses +```shell +awslocal lambda update-function-code --function-name list \ + --s3-bucket hot-reload --s3-key "$(pwd)/lambdas/list" ``` -An alternative is to use a service like MailHog or smtp4dev, and start LocalStack using `SMTP_HOST=host.docker.internal:1025` pointing to the mock SMTP server. +This setup allows you to: -### Run integration tests +- Modify the Lambda handler code in `lambdas/list/handler.py` +- See changes immediately reflected in the UI without rebuilding or redeploying +- Iterate quickly on Lambda logic during development +- Test different response formats or business logic in real-time -Once all resource are created on LocalStack, you can run the automated integration tests. +The hot reloading feature uses LocalStack's ability to watch local file changes and automatically update the Lambda function's code, making development cycles significantly faster than traditional serverless development workflows. -```bash -pytest tests/ -``` - -### GitHub Action +### Integration Testing -The demo LocalStack in CI, `.github/workflows/integration-test.yml` contains a GitHub Action that starts up LocalStack, -deploys the infrastructure to it, and then runs the integration tests. +The sample includes comprehensive integration tests that demonstrate end-to-end testing patterns for serverless applications. The test suite validates the complete workflow from image upload through processing to failure handling. -## Contributing +Key testing scenarios include: -We appreciate your interest in contributing to our project and are always looking for new ways to improve the developer experience. We welcome feedback, bug reports, and even feature ideas from the community. -Please refer to the [contributing file](CONTRIBUTING.md) for more details on how to get started. +- **Image Processing Workflow**: + - Uploads a test image to the source S3 bucket + - Waits for the resize Lambda to process the image (triggered by S3 event notification) + - Verifies the resized image appears in the target bucket + - Confirms the resized image is smaller than the original + - Cleans up test resources +- **Error Handling and Notifications**: + - Uploads a non-image file to trigger Lambda failure + - Verifies that SNS receives the failure notification + - Confirms SES sends an email alert using LocalStack's SES testing endpoint + - Uses LocalStack's `/_aws/ses` endpoint to inspect sent emails without requiring real email infrastructure -## Cloud Pods +The integration tests demonstrate several LocalStack capabilities: -[Cloud Pods](https://docs.localstack.cloud/user-guide/tools/cloud-pods/) are a mechanism that allows you to take a snapshot of the state in your current LocalStack instance, persist it to a storage backend, and easily share it with your team members. +- **S3 Waiters**: Using `s3.get_waiter("object_exists")` to wait for asynchronous processing +- **Lambda Waiters**: Ensuring functions are active before running tests +- **SES Developer Endpoints**: Testing email functionality without external dependencies +- **Event-Driven Testing**: Validating S3 event notifications trigger Lambda functions correctly -You can convert your current AWS infrastructure state to a Cloud Pod using the `localstack` CLI. -Check out our [Getting Started guide](https://docs.localstack.cloud/user-guide/cloud-pods/getting-started/) and [LocalStack Cloud Pods CLI reference](https://docs.localstack.cloud/user-guide/cloud-pods/pods-cli/) to learn more about Cloud Pods and how to use them. +These patterns enable reliable testing of complex serverless applications in isolated local environments. -To inject a Cloud Pod you can use [Cloud Pods Launchpad](https://docs.localstack.cloud/user-guide/cloud-pods/launchpad/) wich quickly injects Cloud Pods into your running LocalStack container. +## Summary -Click here [![LocalStack Pods Launchpad](https://localstack.cloud/gh/launch-pod-badge.svg)](https://app.localstack.cloud/launchpad?url=https://github.com/localstack/sample-serverless-image-resizer-s3-lambda/releases/download/latest/release-pod.zip) to launch the Cloud Pods Launchpad and inject the Cloud Pod for this application by clicking the `Inject` button. +This sample application demonstrates how to build, deploy, and test a complete serverless image processing workflow using LocalStack. It showcases the following patterns: +- Building event-driven serverless applications with S3, Lambda, and SNS integration +- Using S3 pre-signed URLs for direct client-side uploads +- Implementing error handling with SNS and SES notifications +- Leveraging Lambda function URLs for HTTP-triggered functions +- Utilizing LocalStack's hot reloading for rapid development cycles +- Writing comprehensive integration tests for serverless applications +- Using LocalStack's developer endpoints for testing email functionality -Alternatively, you can inject the pod by using the `localstack` CLI. -First, you need to download the pod you want to inject from the [releases](https://github.com/localstack/sample-serverless-image-resizer-s3-lambda/releases). -Then run: +## Learn More -```sh -localstack state import /path/to/release-pod.zip -``` +- [LocalStack Lambda Hot Reloading](https://docs.localstack.cloud/aws/tooling/lambda-tools/hot-reloading/) +- [S3 Event Notifications](https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html) +- [Lambda Function URLs](http://docs.aws.amazon.com/lambda/latest/dg/urls-configuration.html) +- [LocalStack Internal Endpoints](https://docs.localstack.cloud/aws/capabilities/networking/internal-endpoints/) +- [AWS SDK for Python](https://docs.localstack.cloud/aws/integrations/aws-sdks/python-boto3/)