/processing-s3-uploads-with-step-functions

name: processing-s3-uploads-with-step-functions
description: >
  Deploy an event-driven workflow that routes S3 uploads to either Lambda or Fargate
  via Step Functions based on file size. Uses EventBridge to trigger a Step Functions
  state machine when objects are uploaded to S3. Small files are processed by Lambda,
  large files by a Fargate task. Includes VPC, ECR repository, ECS cluster, and scoped
  IAM roles. Trigger keywords: Step Functions, Fargate, Lambda, S3 event, EventBridge,
  ECS, ECR, file processing, workflow orchestration, serverless.
version: 1

Step Functions Workflow: Route S3 Uploads to Lambda or Fargate

Overview

This skill deploys an event-driven workflow using AWS CLI. When a file is uploaded to an S3 bucket, EventBridge triggers a Step Functions state machine. The state machine checks the file size and routes processing to either a Lambda function (files ≤ 6 MB) or a Fargate task (files > 6 MB).

The architecture includes:

• An S3 bucket with EventBridge notifications enabled
• An EventBridge rule that triggers Step Functions on S3 object creation
• A Step Functions state machine with a Choice state for routing
• A Lambda function for processing small files
• An ECS Fargate task for processing large files
• A VPC with two subnets, internet gateway, and security group
• An ECR repository for the Fargate container image
• Scoped IAM roles for Lambda, Step Functions, and ECS tasks

Use this skill when:

• You need to process S3 uploads with different compute based on file size
• You want a serverless workflow that can handle both small and large files
• You need Step Functions orchestration with Lambda and Fargate

Do not use this skill when:

• All files are small enough for Lambda (use S3 → Lambda directly)
• You need real-time streaming (use Kinesis)
• You don't need file-size-based routing

Prerequisites

1. **AWS CLI v2** — Installed and configured. Verify with `aws sts get-caller-identity`. 2. **Python 3.12** — For the Lambda function runtime. 3. **Docker** — For building and pushing the Fargate container image.

Parameters

• bucket_name (required): Name for the S3 bucket (globally unique, lowercase, 3-63 characters)
• region (required): AWS region for all resources
• ecr_repo_name (required): Name for the ECR repository
• state_machine_name (required): Name for the Step Functions state machine
• kms_key_arn (optional): ARN of a KMS key for CloudWatch Logs encryption. If not provided, create one with `aws kms create-key --description "Key for CloudWatch Logs encryption" --region {region}`

Constraints for parameter acquisition:

• You MUST ask for all required parameters upfront in a single prompt
• You MUST support multiple input methods (direct input, file path, URL)
• You MUST confirm successful acquisition of all parameters before proceeding
• You MUST validate that bucket_name follows S3 naming rules

Procedures

Step 0: Verify Dependencies

Constraints:

• You MUST verify the following tools are available: aws-cli, python3 (3.12+), docker
• You MUST inform the user about any missing tools with a clear message
• You MUST ask if the user wants to proceed despite missing tools
• You MUST respect the customer's decision to abort at any point
• You MUST explain to the customer what step is being executed, why, and which tool is being called

Step 1: Retrieve AWS Account ID

Constraints:

• You MUST retrieve the account ID with: `aws sts get-caller-identity --query 'Account' --output text`
• You MUST store the result as {account_id} for use in all subsequent steps
• You MUST abort if credentials are not configured

Step 2: Get the Default VPC and Networking

Constraints:

• You MUST retrieve the default VPC ID with:

`aws ec2 describe-vpcs --filters Name=isDefault,Values=true --query 'Vpcs[0].VpcId' --output text --region {region}`

• If no default VPC exists, inform the user they must create one with `aws ec2 create-default-vpc --region {region}` or provide a VPC ID manually
• You MUST retrieve two subnet IDs from the default VPC:

`aws ec2 describe-subnets --filters Name=vpc-id,Values={vpc_id} --query 'Subnets[0:2].SubnetId' --output text --region {region}`

• You MUST create a security group in the default VPC:

`aws ec2 create-security-group --group-name fargate-sg --description "Security group for Fargate tasks" --vpc-id {vpc_id} --region {region}`

• You MUST configure security group egress rules to allow only HTTPS and DNS outbound. First revoke the default allow-all egress rule:

`aws ec2 revoke-security-group-egress --group-id {sg_id} --ip-permissions IpProtocol=-1,IpRanges='[{CidrIp=0.0.0.0/0}]' --region {region}` Then add scoped rules: `aws ec2 authorize-security-group-egress --group-id {sg_id} --protocol tcp --port 443 --cidr 0.0.0.0/0 --region {region}` and `aws ec2 authorize-security-group-egress --group-id {sg_id} --protocol udp --port 53 --cidr 0.0.0.0/0 --region {region}`

• You MUST recommend VPC endpoints for S3 and CloudWatch Logs for production workloads to avoid internet-routed traffic and eliminate the need for broad egress rules
• You MUST capture {vpc_id}, {subnet1_id}, {subnet2_id}, and {sg_id} for use in later steps

Step 3: Create the ECR Repository

Constraints:

• You MUST create the repository with:

`aws ecr create-repository --repository-name {ecr_repo_name} --region {region}`

• You MUST capture the repositoryUri from the response

Step 4: Build and Push the Container Image

Constraints:

• You MUST verify Docker is installed by running `docker --version`. If Docker is not installed, instruct the user to install it from https://docs.docker.com/get-docker/ and abort until it is available
• You MUST authenticate Docker with ECR:

`aws ecr get-login-password --region {region} | docker login --username AWS --password-stdin {account_id}.dkr.ecr.{region}.amazonaws.com`

• The Dockerfile and processor code are in `scripts/Dockerfile` and `scripts/fargate_processor.py`
• You MUST bui