Amazon S3 Configuration

Connect to Amazon S3 for file storage, data lakes, backups, and object storage needs.

Connection Parameters

Required Fields

• Bucket Name: S3 bucket name (e.g., my-data-bucket)
• Region: AWS region (e.g., us-east-1)
• Access Key ID: AWS access key
• Secret Access Key: AWS secret key

Optional Fields

• Custom Endpoint: Optional (for S3-compatible services like MinIO, DigitalOcean Spaces)

Configuration Example

When creating an S3 data source, provide the following information:

Field	Example Value	Notes
Name	prod-s3-storage	Unique identifier
Label	Production S3 Storage	Display name
Bucket Name	my-data-bucket	S3 bucket name
Region	us-east-1	AWS region
Access Key ID	AKIA...	AWS access key
Secret Access Key	••••••••	Encrypted at rest
Custom Endpoint	-	For S3-compatible services

AWS IAM Configuration

Creating an IAM User

1. Go to AWS IAM Console
2. Navigate to Users → Add Users
3. Select Access key - Programmatic access
4. Attach policies (see Required Permissions)
5. Complete user creation
6. Save the Access Key ID and Secret Access Key

Security

Save your secret access key immediately. AWS won't show it again after creation.

Required Permissions

Grant the following permissions to your IAM user:

Read-Only Access

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:GetObjectVersion",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::my-data-bucket",
        "arn:aws:s3:::my-data-bucket/*"
      ]
    }
  ]
}

Read-Write Access

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:GetObjectVersion",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::my-data-bucket",
        "arn:aws:s3:::my-data-bucket/*"
      ]
    }
  ]
}

Full Access (Including Bucket Management)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::my-data-bucket",
        "arn:aws:s3:::my-data-bucket/*"
      ]
    }
  ]
}

Least Privilege

Use the minimum required permissions. For most applications, read-only or read-write access is sufficient.

Usage in Applications

Python Example (boto3)

import os, json
import boto3

# Parse STRONGLY_SERVICES environment variable
services = json.loads(os.getenv('STRONGLY_SERVICES', '{}'))
datasources = services.get('datasources', {})

# Get S3 data source
s3_config = datasources['prod-s3-storage']
creds = s3_config['credentials']

# Create S3 client
s3 = boto3.client(
    's3',
    aws_access_key_id=creds['access_key_id'],
    aws_secret_access_key=creds['secret_access_key'],
    region_name=creds['region']
)

# List objects in bucket
response = s3.list_objects_v2(Bucket=creds['bucket_name'])
for obj in response.get('Contents', []):
    print(f"File: {obj['Key']}, Size: {obj['Size']} bytes")

# Upload a file
s3.upload_file('local-file.txt', creds['bucket_name'], 'remote-file.txt')

# Download a file
s3.download_file(creds['bucket_name'], 'remote-file.txt', 'downloaded-file.txt')

# Read file content directly
obj = s3.get_object(Bucket=creds['bucket_name'], Key='data.json')
content = obj['Body'].read().decode('utf-8')
print(content)

Python with Custom Endpoint (MinIO)

import boto3

# For S3-compatible services like MinIO
s3 = boto3.client(
    's3',
    endpoint_url=creds.get('endpoint_url', None),  # e.g., 'https://minio.example.com'
    aws_access_key_id=creds['access_key_id'],
    aws_secret_access_key=creds['secret_access_key'],
    region_name=creds['region']
)

Node.js Example (AWS SDK v3)

const { S3Client, ListObjectsV2Command, GetObjectCommand, PutObjectCommand } = require('@aws-sdk/client-s3');
const { createReadStream, createWriteStream } = require('fs');

// Parse STRONGLY_SERVICES environment variable
const services = JSON.parse(process.env.STRONGLY_SERVICES || '{}');
const datasources = services.datasources || {};

// Get S3 data source
const s3Config = datasources['prod-s3-storage'];
const creds = s3Config.credentials;

// Create S3 client
const s3Client = new S3Client({
  region: creds.region,
  credentials: {
    accessKeyId: creds.access_key_id,
    secretAccessKey: creds.secret_access_key
  }
});

// List objects
const listCommand = new ListObjectsV2Command({
  Bucket: creds.bucket_name
});
const listResponse = await s3Client.send(listCommand);
console.log('Objects:', listResponse.Contents);

// Upload a file
const uploadCommand = new PutObjectCommand({
  Bucket: creds.bucket_name,
  Key: 'remote-file.txt',
  Body: createReadStream('local-file.txt')
});
await s3Client.send(uploadCommand);

// Download a file
const downloadCommand = new GetObjectCommand({
  Bucket: creds.bucket_name,
  Key: 'remote-file.txt'
});
const downloadResponse = await s3Client.send(downloadCommand);
const body = await downloadResponse.Body.transformToString();
console.log(body);

Go Example

package main

import (
    "context"
    "encoding/json"
    "fmt"
    "os"

    "github.com/aws/aws-sdk-go-v2/aws"
    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/credentials"
    "github.com/aws/aws-sdk-go-v2/service/s3"
)

func main() {
    // Parse STRONGLY_SERVICES environment variable
    var services map[string]interface{}
    json.Unmarshal([]byte(os.Getenv("STRONGLY_SERVICES")), &services)

    datasources := services["datasources"].(map[string]interface{})
    s3Config := datasources["prod-s3-storage"].(map[string]interface{})
    creds := s3Config["credentials"].(map[string]interface{})

    // Create AWS config
    cfg, err := config.LoadDefaultConfig(context.TODO(),
        config.WithRegion(creds["region"].(string)),
        config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider(
            creds["access_key_id"].(string),
            creds["secret_access_key"].(string),
            "",
        )),
    )
    if err != nil {
        panic(err)
    }

    // Create S3 client
    client := s3.NewFromConfig(cfg)

    // List objects
    output, err := client.ListObjectsV2(context.TODO(), &s3.ListObjectsV2Input{
        Bucket: aws.String(creds["bucket_name"].(string)),
    })
    if err != nil {
        panic(err)
    }

    for _, object := range output.Contents {
        fmt.Printf("File: %s, Size: %d bytes\n", *object.Key, object.Size)
    }
}

Common Operations

Upload File with Metadata

s3.upload_file(
    'local-file.txt',
    creds['bucket_name'],
    'remote-file.txt',
    ExtraArgs={
        'Metadata': {
            'uploaded-by': 'my-app',
            'content-type': 'text/plain'
        },
        'ContentType': 'text/plain'
    }
)

Generate Presigned URL

Create temporary URLs for file access:

# Generate presigned URL (valid for 1 hour)
url = s3.generate_presigned_url(
    'get_object',
    Params={
        'Bucket': creds['bucket_name'],
        'Key': 'private-file.txt'
    },
    ExpiresIn=3600
)
print(f"Temporary URL: {url}")

Copy Objects Between Buckets

copy_source = {
    'Bucket': 'source-bucket',
    'Key': 'source-file.txt'
}

s3.copy_object(
    CopySource=copy_source,
    Bucket='destination-bucket',
    Key='destination-file.txt'
)

Delete Objects

# Delete single object
s3.delete_object(Bucket=creds['bucket_name'], Key='file-to-delete.txt')

# Delete multiple objects
s3.delete_objects(
    Bucket=creds['bucket_name'],
    Delete={
        'Objects': [
            {'Key': 'file1.txt'},
            {'Key': 'file2.txt'},
            {'Key': 'file3.txt'}
        ]
    }
)

S3-Compatible Services

This configuration also works with S3-compatible services:

MinIO

s3 = boto3.client(
    's3',
    endpoint_url='https://minio.example.com',
    aws_access_key_id=creds['access_key_id'],
    aws_secret_access_key=creds['secret_access_key'],
    region_name='us-east-1'
)

DigitalOcean Spaces

s3 = boto3.client(
    's3',
    endpoint_url='https://nyc3.digitaloceanspaces.com',
    aws_access_key_id=creds['access_key_id'],
    aws_secret_access_key=creds['secret_access_key'],
    region_name='nyc3'
)

Wasabi

s3 = boto3.client(
    's3',
    endpoint_url='https://s3.wasabisys.com',
    aws_access_key_id=creds['access_key_id'],
    aws_secret_access_key=creds['secret_access_key'],
    region_name='us-east-1'
)

Common Issues

Access Denied

• Verify IAM permissions include required S3 actions
• Check bucket policy allows access from IAM user
• Ensure bucket exists and name is correct
• Verify access keys are correct and not expired

Invalid Access Key ID

• Check access key ID is correct
• Verify IAM user still exists
• Ensure access key hasn't been deleted or deactivated
• Regenerate keys if necessary

Bucket Not Found

• Verify bucket name is correct (case-sensitive)
• Check bucket exists in correct region
• Ensure IAM user has permission to access bucket

Region Mismatch

• Verify region matches bucket location
• Use us-east-1 for buckets without specific region
• Check region when creating S3 client

Best Practices

1. Use IAM Roles: For applications running on AWS, use IAM roles instead of access keys
2. Least Privilege: Grant minimal required permissions
3. Enable Versioning: Use S3 versioning for important data
4. Server-Side Encryption: Enable encryption at rest for sensitive data
5. Lifecycle Policies: Configure lifecycle policies to manage storage costs
6. Access Logging: Enable S3 access logging for audit trails
7. Secure Keys: Never commit access keys to version control
8. Use HTTPS: Always use HTTPS endpoints for data in transit encryption
9. Multipart Upload: Use multipart upload for large files (>100MB)
10. Monitor Costs: Set up billing alerts and monitor S3 usage

Performance Optimization

Multipart Upload for Large Files

import boto3
from boto3.s3.transfer import TransferConfig

# Configure multipart upload thresholds
config = TransferConfig(
    multipart_threshold=1024 * 25,  # 25 MB
    max_concurrency=10,
    multipart_chunksize=1024 * 25,
    use_threads=True
)

s3.upload_file(
    'large-file.zip',
    creds['bucket_name'],
    'remote-large-file.zip',
    Config=config
)

Parallel Downloads

# Download multiple files in parallel
from concurrent.futures import ThreadPoolExecutor

def download_file(key):
    s3.download_file(creds['bucket_name'], key, f"local-{key}")

files = ['file1.txt', 'file2.txt', 'file3.txt']
with ThreadPoolExecutor(max_workers=5) as executor:
    executor.map(download_file, files)

Related Resources

• Data Sources Overview
• How to Connect Data Sources
• AWS S3 Documentation
• AWS S3 Best Practices
• boto3 Documentation