Simple Storage Service (S3)

Simple Storage Service is the most-used service in all of AWS - an object store with 11 nines of durability, infinite capacity, and a price measured in cents per GB. It has evolved into a platform for analytics, AI workloads, and file-system access.

Introduction

S3 stores arbitrary files (called objects) inside containers called buckets. Unlike a file system, S3 has no real directory hierarchy - the "folders" you see in the console are just key prefixes. S3 is used for static assets, backups, data lake raw storage, Lambda deployment packages, CloudFront origins, and much more. Recent additions like S3 Tables, S3 Vectors, and S3 Files extend it into analytics, AI, and POSIX workloads.

Quick Navigation

1. Buckets & Objects

Core concepts and CLI ops

2. Storage Classes

Cost vs retrieval tradeoffs

3. Versioning & Lifecycle

Version history and expiry rules

4. Presigned URLs

Time-limited private access

5. Bucket Policies

Access control and static hosting

6. S3 File System

NFS mount on top of S3

7. Storage Lens

Org-wide metrics and cost insights

8. Modern Capabilities

Tables, Vectors, Metadata

1. Buckets and Objects

Buckets
  • Globally unique names across all AWS accounts
  • Tied to a specific region (data stays there unless replicated)
  • No limit on number of objects inside
  • Max 100 buckets per account (soft limit, can request more)
Objects
  • Max object size: 5 TB (use multipart upload above 100 MB)
  • Key is the full path: logs/2026/01/app.log
  • Each object can have up to 10 user-defined metadata tags
  • Objects are immutable - you cannot modify them in place; a write to the same key replaces the object (with versioning enabled, the old copy is preserved as a prior version)
# Create a bucket (bucket names are globally unique)
aws s3 mb s3://my-app-bucket-2026 --region us-east-1

# Upload a file
aws s3 cp ./app.zip s3://my-app-bucket-2026/releases/app.zip

# Upload an entire directory recursively
aws s3 sync ./dist s3://my-app-bucket-2026/web/ --delete

# Download a file
aws s3 cp s3://my-app-bucket-2026/releases/app.zip ./app.zip

# List objects with their sizes
aws s3 ls s3://my-app-bucket-2026/ --recursive --human-readable

# Delete an object
aws s3 rm s3://my-app-bucket-2026/releases/old-app.zip

2. Storage Classes

S3 offers multiple storage classes with different pricing trade-offs on storage cost vs retrieval cost:

ClassUse CaseRetrievalRelative Cost
STANDARDFrequently accessed dataMillisecondsHighest
STANDARD_IAInfrequently accessed, but fast when neededMilliseconds~40% cheaper storage, retrieval fee
INTELLIGENT_TIERINGUnknown or changing access patternsMillisecondsSmall monitoring fee, auto-tiers
GLACIER_IRArchives needing occasional retrievalMillisecondsVery cheap storage
GLACIERLong-term archives (compliance, backups)Minutes to hoursCheapest storage

3. Versioning and Lifecycle Policies

Enable versioning on a bucket to keep every historical version of every object. Deleting a versioned object places a delete marker rather than removing it permanently, making accidental deletes recoverable. Combine versioning with lifecycle policies to automatically move or expire old versions:

# lifecycle.json
{
  "Rules": [
    {
      "ID": "move-old-logs",
      "Status": "Enabled",
      "Filter": { "Prefix": "logs/" },
      "Transitions": [
        { "Days": 30,  "StorageClass": "STANDARD_IA" },
        { "Days": 90,  "StorageClass": "GLACIER" }
      ],
      "Expiration": { "Days": 365 }
    }
  ]
}

# Apply it
aws s3api put-bucket-lifecycle-configuration \
  --bucket my-app-bucket-2026 \
  --lifecycle-configuration file://lifecycle.json

4. Presigned URLs

A presigned URL grants time-limited access to a private S3 object without changing bucket permissions. Your server generates it using its own IAM credentials, then hands the URL to the client. Common uses: letting users download their own files, or uploading directly to S3 from a browser (presigned PUT URL) without routing the upload through your server.

# Generate a presigned URL (expires in 3600 seconds = 1 hour)
aws s3 presign s3://my-app-bucket-2026/reports/invoice.pdf \
  --expires-in 3600

# Returns a URL like:
# https://my-app-bucket-2026.s3.amazonaws.com/reports/invoice.pdf?X-Amz-Signature=...
# Anyone with this URL can download the object until it expires

# Python equivalent using boto3
import boto3
s3 = boto3.client("s3")
url = s3.generate_presigned_url(
    "get_object",
    Params={"Bucket": "my-app-bucket-2026", "Key": "reports/invoice.pdf"},
    ExpiresIn=3600,
)

5. Bucket Policies and Static Website Hosting

To serve a React/Vue/static site directly from S3, enable static website hosting and apply a bucket policy that allows public reads. For production, put CloudFront in front to add HTTPS, caching, and a custom domain:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PublicReadGetObject",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::my-static-site/*"
    }
  ]
}

# Apply the policy
aws s3api put-bucket-policy \
  --bucket my-static-site \
  --policy file://bucket-policy.json

Security note: public bucket policies expose all matching objects to the internet. Only use them for intentionally public assets. Keep private data in buckets with no public access and use presigned URLs or IAM roles to control access.

6. S3 File System (S3 Files)

ML training pipelines, HPC jobs, and scientific tools often require a POSIX file system rather than an object API. The traditional workaround is to copy data out of S3 into EFS or FSx before processing, then copy results back. S3 Files eliminates this by mounting the bucket itself as an NFS file system, your application reads and writes normally, and the data never leaves S3.

How it works
  • Built on Amazon EFS internals - active data is lazily loaded onto high-performance storage; the authoritative copy always lives in S3
  • Writes go to durable storage first, then sync back to S3 automatically
  • Unused data expires on a configurable window (1-365 days, default 30), keeping the hot tier lean
  • Up to 25,000 concurrent connections across any AWS compute (EC2, ECS, Lambda, SageMaker)
  • Dual access - the same bucket is simultaneously accessible via file system and standard S3 API
Best suited for
  • ML training pipelines and data preparation that expect a POSIX file system
  • AI agent coordination with shared workspaces across many compute nodes
  • Scientific and bioinformatics workflows using legacy file-based tools
  • Team collaboration on large shared datasets without duplicating data into EFS
Key numbers: 10M+ IOPS per bucket, 4 TB/s+ aggregate read throughput, up to 90% lower cost compared to cycling data between S3 and a separate file system.
# Pre-requisite: create a File System access point in the S3 Console.
# The NFS hostname is displayed in the console once the access point is ready.

# ---- EC2 / ECS host: mount via NFS4 ----
sudo mount -t nfs4 \
  -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600 \
  <nfs-hostname-from-console>.s3-accesspoint.us-east-1.amazonaws.com:/ \
  /mnt/s3files

# Standard POSIX operations - no application code changes needed
ls /mnt/s3files/training-data/
cp ./model_v2.bin /mnt/s3files/models/
cat /mnt/s3files/config/settings.json

# Objects written via NFS are immediately accessible via the S3 API
aws s3 ls s3://my-ml-bucket/models/

S3 Files: Dual-Access Architecture

AWS COMPUTEEC2EC2 InstanceML / HPC workloadsFGECS / Fargatecontainerized appsλLambdaserverless functionsS3S3 BucketS3 Files enabledEXTS3 API / CLIboto3, SDK, CLINFS mountNFS mountNFS mountS3 API

Compute nodes mount the bucket as a POSIX file system; existing S3 tooling continues to work unchanged via the object API.

When not to use it: if your pipeline already uses the S3 API naturally, stick with it. S3 Files targets workloads that today copy data out of S3 into EFS or FSx just to get file semantics. Pay-as-you-go with no provisioned capacity required.

7. S3 Storage Lens

Storage Lens is an analytics dashboard that aggregates storage metrics across every bucket in your account, or across all accounts in an AWS Organization. Instead of clicking into each bucket individually, you get a unified view of usage trends, activity rates, and cost-optimization recommendations in one place.

FeatureFree TierAdvanced Tier
CostFree~$0.20 / million objects / month
Metrics28 usage metrics35+ including activity metrics
ScopeSingle accountMulti-account / Organization
RecommendationsBasicDetailed cost-optimization suggestions
CloudWatch publishNoYes
Prefix-level breakdownNoYes (configurable depth)

The most actionable metrics for cost control are:

Non-current version storage

Bytes held by old object versions when versioning is enabled. Usually safe to expire after a retention window.

Incomplete multipart uploads

Parts from aborted uploads that accumulate silently. Add a lifecycle rule to abort incomplete MPUs after 7 days.

Objects not accessed in 90+ days

Storage Lens surfaces these buckets as Intelligent-Tiering candidates, saving up to 40% on storage cost.

# Create a Storage Lens dashboard with Advanced metrics enabled
aws s3control put-storage-lens-configuration \
  --account-id 123456789012 \
  --config-id my-org-lens \
  --storage-lens-configuration file://lens-config.json

# lens-config.json
{
  "Id": "my-org-lens",
  "IsEnabled": true,
  "AccountLevel": {
    "ActivityMetrics": { "IsEnabled": true },
    "BucketLevel": {
      "ActivityMetrics": { "IsEnabled": true },
      "PrefixLevel": {
        "StorageMetrics": {
          "IsEnabled": true,
          "SelectionCriteria": { "MaxDepth": 2, "MinStorageBytes": 1048576 }
        }
      }
    }
  },
  "DataExport": {
    "S3BucketDestination": {
      "Format": "Parquet",
      "OutputSchemaVersion": "V_1",
      "AccountId": "123456789012",
      "Arn": "arn:aws:s3:::my-lens-exports"
    }
  }
}

Default dashboard: Storage Lens is enabled automatically at the account level with 28 free usage metrics. Check it in the S3 console under Storage Lens → Dashboards. Enable the Advanced tier only if you need per-prefix activity breakdowns or cross-account aggregation.

8. Modern S3 Capabilities

Over the past years AWS has extended S3 beyond raw object storage into structured data, vector search, and automatic metadata cataloging. These three capabilities reduce the need for separate purpose-built services in many common workflows.

8a. S3 Tables

S3 Tables is a new bucket type that stores data in Apache Iceberg format and manages it for you. AWS runs compaction, snapshot cleanup, and concurrent write coordination automatically - no Glue catalog setup, no Spark vacuum jobs, no manual partition management.

  • What it stores: structured tabular data as Parquet files with Iceberg metadata
  • AWS manages: automatic compaction, snapshot expiry, concurrent write coordination (optimistic concurrency)
  • Query engines: Athena, EMR (Spark / Hive), Redshift Spectrum - all read natively with no extra setup
  • vs DIY Iceberg on S3: eliminates compaction jobs, Glue catalog maintenance, and vacuum scripts
  • Best for: data lake tables that multiple analytics engines need to query and write concurrently
import boto3

# Create an S3 Tables bucket - a specialized bucket for structured tabular data
tables = boto3.client("s3tables", region_name="us-east-1")
bucket = tables.create_table_bucket(name="analytics-tables")
bucket_arn = bucket["arn"]

# Create a namespace (logical grouping, similar to a database schema)
tables.create_namespace(tableBucketARN=bucket_arn, namespace=["sales"])

# Create an Iceberg table - AWS handles compaction and snapshots automatically
tables.create_table(
    tableBucketARN=bucket_arn,
    namespace="sales",
    name="orders",
    format="ICEBERG",
)

# Query directly from Athena - no Glue catalog setup required
# (catalog format: "s3tablescatalog/<bucket-name>"."<namespace>"."<table>")
# SELECT customer_id, SUM(total_amount) AS revenue
# FROM "s3tablescatalog/analytics-tables"."sales"."orders"
# WHERE order_date >= DATE '2026-01-01'
# GROUP BY customer_id
# ORDER BY revenue DESC;

8b. S3 Vectors

S3 Vectors (preview, 2025) adds a vector index directly to S3. Store high-dimensional embeddings alongside your objects and run approximate nearest-neighbor (ANN) search without spinning up a separate vector database.

  • How it works: create an index (specifying dimension and distance metric), then put vectors with metadata; query returns the top-K most similar vectors
  • Distance metrics: cosine similarity, euclidean distance, dot product
  • Use cases: semantic document search, image similarity, RAG pipelines, recommendation systems
  • Replaces: a separate Pinecone, OpenSearch k-NN, or pgvector deployment for many workloads
  • Limits: suited for hundreds of millions of vectors; for sub-10 ms latency at extreme scale, a purpose-built vector DB still wins
import boto3

client = boto3.client("s3vectors", region_name="us-east-1")

# Create a vector index - dimension must match your embedding model
client.create_index(
    vectorBucketName="my-vectors-bucket",
    indexName="doc-embeddings",
    dataType="float32",
    dimension=1536,         # matches OpenAI text-embedding-3-small
    distanceMetric="cosine",
)

# Store a vector alongside metadata
client.put_vectors(
    vectorBucketName="my-vectors-bucket",
    indexName="doc-embeddings",
    vectors=[{
        "key": "reports/q1-2026.pdf",
        "data": {"float32": embedding_array},   # your 1536-float list
        "metadata": {"department": "finance", "year": "2026"}
    }]
)

# Nearest-neighbor search - no separate vector DB required
results = client.query_vectors(
    vectorBucketName="my-vectors-bucket",
    indexName="doc-embeddings",
    queryVector={"float32": query_embedding},
    topK=5,
)
for match in results["vectors"]:
    print(match["key"], round(match["score"], 4), match["metadata"])

8c. S3 Metadata

S3 Metadata (2025) automatically extracts and indexes object metadata as objects land in a bucket. The index is stored in an S3 Tables bucket and is queryable with Athena using SQL - no object downloads or bucket scans required.

  • What gets indexed: system metadata (size, ETag, last-modified, content-type) plus all user-defined tags
  • Real-time: metadata is indexed as objects are written; unlike S3 Inventory (daily batch), queries reflect the current state
  • Zero pipeline changes: enable it on an existing bucket and S3 backfills metadata for all existing objects
  • Built on S3 Tables: the metadata table is an Iceberg table you can join against your own data tables
  • Example query: find all PDFs tagged department=finance uploaded in the last 7 days without touching any object
import boto3

# Step 1: create a metadata table bucket once per region (uses S3 Tables under the hood)
tables = boto3.client("s3tables", region_name="us-east-1")
meta_bucket = tables.create_table_bucket(name="s3-metadata-store")

# Step 2: enable automatic metadata indexing on your data bucket
s3 = boto3.client("s3")
s3.put_bucket_metadata_table_configuration(
    Bucket="my-app-bucket-2026",
    MetadataTableConfiguration={
        "S3TablesDestination": {
            "TableBucketARN": meta_bucket["arn"],
            "TableName": "my-app-bucket-2026"
        }
    }
)
# S3 now indexes every object as it lands - no pipeline changes needed

# Query metadata via Athena: find recent PDFs tagged department=finance
# (verify the exact namespace in the S3 Tables console after enabling metadata)
# SELECT key, size, last_modified
# FROM "s3tablescatalog/s3-metadata-store"."aws_s3_metadata"."my-app-bucket-2026"
# WHERE content_type = 'application/pdf'
#   AND json_extract_scalar(user_metadata, '$.department') = 'finance'
#   AND last_modified >= NOW() - INTERVAL '7' DAY;

Key Takeaways

  • Buckets are globally unique containers; objects inside have no real directory structure, just key prefixes
  • Storage classes trade storage cost for retrieval speed/cost; use INTELLIGENT_TIERING when access patterns are unknown
  • Versioning protects against accidental deletes and overwrites; use with lifecycle rules to expire old versions
  • Lifecycle policies automatically tier or expire data, saving significant costs for log archives
  • Presigned URLs grant time-limited access to private objects; use for user downloads and browser-side uploads
  • Static hosting - pair S3 with CloudFront for HTTPS, caching, and a custom domain on static sites
  • S3 Files mounts a bucket as an NFS file system; eliminates data duplication for ML and file-based workloads that need POSIX access
  • Storage Lens provides org-wide metrics and cost-saving recommendations; free tier covers basic usage and is enabled by default
  • S3 Tables adds native Apache Iceberg table support with automatic compaction; query with Athena, EMR, or Redshift without managing a Glue catalog
  • S3 Vectors stores and queries vector embeddings directly in S3 using ANN search; reduces need for a separate vector database in RAG and semantic search workloads
  • S3 Metadata automatically indexes every object as it lands and exposes a live SQL-queryable metadata table; replaces manual bucket scanning for object discovery