Simple Storage Service (S3)
Simple Storage Service is the most-used service in all of AWS - an object store with 11 nines of durability, infinite capacity, and a price measured in cents per GB. It has evolved into a platform for analytics, AI workloads, and file-system access.
Introduction
S3 stores arbitrary files (called objects) inside containers called buckets. Unlike a file system, S3 has no real directory hierarchy - the "folders" you see in the console are just key prefixes. S3 is used for static assets, backups, data lake raw storage, Lambda deployment packages, CloudFront origins, and much more. Recent additions like S3 Tables, S3 Vectors, and S3 Files extend it into analytics, AI, and POSIX workloads.
Quick Navigation
1. Buckets & Objects
Core concepts and CLI ops
2. Storage Classes
Cost vs retrieval tradeoffs
3. Versioning & Lifecycle
Version history and expiry rules
4. Presigned URLs
Time-limited private access
5. Bucket Policies
Access control and static hosting
6. S3 File System
NFS mount on top of S3
7. Storage Lens
Org-wide metrics and cost insights
8. Modern Capabilities
Tables, Vectors, Metadata
1. Buckets and Objects
Buckets
- Globally unique names across all AWS accounts
- Tied to a specific region (data stays there unless replicated)
- No limit on number of objects inside
- Max 100 buckets per account (soft limit, can request more)
Objects
- Max object size: 5 TB (use multipart upload above 100 MB)
- Key is the full path:
logs/2026/01/app.log - Each object can have up to 10 user-defined metadata tags
- Objects are immutable - you cannot modify them in place; a write to the same key replaces the object (with versioning enabled, the old copy is preserved as a prior version)
# Create a bucket (bucket names are globally unique) aws s3 mb s3://my-app-bucket-2026 --region us-east-1 # Upload a file aws s3 cp ./app.zip s3://my-app-bucket-2026/releases/app.zip # Upload an entire directory recursively aws s3 sync ./dist s3://my-app-bucket-2026/web/ --delete # Download a file aws s3 cp s3://my-app-bucket-2026/releases/app.zip ./app.zip # List objects with their sizes aws s3 ls s3://my-app-bucket-2026/ --recursive --human-readable # Delete an object aws s3 rm s3://my-app-bucket-2026/releases/old-app.zip
2. Storage Classes
S3 offers multiple storage classes with different pricing trade-offs on storage cost vs retrieval cost:
| Class | Use Case | Retrieval | Relative Cost |
|---|---|---|---|
STANDARD | Frequently accessed data | Milliseconds | Highest |
STANDARD_IA | Infrequently accessed, but fast when needed | Milliseconds | ~40% cheaper storage, retrieval fee |
INTELLIGENT_TIERING | Unknown or changing access patterns | Milliseconds | Small monitoring fee, auto-tiers |
GLACIER_IR | Archives needing occasional retrieval | Milliseconds | Very cheap storage |
GLACIER | Long-term archives (compliance, backups) | Minutes to hours | Cheapest storage |
3. Versioning and Lifecycle Policies
Enable versioning on a bucket to keep every historical version of every object. Deleting a versioned object places a delete marker rather than removing it permanently, making accidental deletes recoverable. Combine versioning with lifecycle policies to automatically move or expire old versions:
# lifecycle.json
{
"Rules": [
{
"ID": "move-old-logs",
"Status": "Enabled",
"Filter": { "Prefix": "logs/" },
"Transitions": [
{ "Days": 30, "StorageClass": "STANDARD_IA" },
{ "Days": 90, "StorageClass": "GLACIER" }
],
"Expiration": { "Days": 365 }
}
]
}
# Apply it
aws s3api put-bucket-lifecycle-configuration \
--bucket my-app-bucket-2026 \
--lifecycle-configuration file://lifecycle.json4. Presigned URLs
A presigned URL grants time-limited access to a private S3 object without changing bucket permissions. Your server generates it using its own IAM credentials, then hands the URL to the client. Common uses: letting users download their own files, or uploading directly to S3 from a browser (presigned PUT URL) without routing the upload through your server.
# Generate a presigned URL (expires in 3600 seconds = 1 hour)
aws s3 presign s3://my-app-bucket-2026/reports/invoice.pdf \
--expires-in 3600
# Returns a URL like:
# https://my-app-bucket-2026.s3.amazonaws.com/reports/invoice.pdf?X-Amz-Signature=...
# Anyone with this URL can download the object until it expires
# Python equivalent using boto3
import boto3
s3 = boto3.client("s3")
url = s3.generate_presigned_url(
"get_object",
Params={"Bucket": "my-app-bucket-2026", "Key": "reports/invoice.pdf"},
ExpiresIn=3600,
)5. Bucket Policies and Static Website Hosting
To serve a React/Vue/static site directly from S3, enable static website hosting and apply a bucket policy that allows public reads. For production, put CloudFront in front to add HTTPS, caching, and a custom domain:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "PublicReadGetObject",
"Effect": "Allow",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::my-static-site/*"
}
]
}
# Apply the policy
aws s3api put-bucket-policy \
--bucket my-static-site \
--policy file://bucket-policy.jsonSecurity note: public bucket policies expose all matching objects to the internet. Only use them for intentionally public assets. Keep private data in buckets with no public access and use presigned URLs or IAM roles to control access.
6. S3 File System (S3 Files)
ML training pipelines, HPC jobs, and scientific tools often require a POSIX file system rather than an object API. The traditional workaround is to copy data out of S3 into EFS or FSx before processing, then copy results back. S3 Files eliminates this by mounting the bucket itself as an NFS file system, your application reads and writes normally, and the data never leaves S3.
How it works
- Built on Amazon EFS internals - active data is lazily loaded onto high-performance storage; the authoritative copy always lives in S3
- Writes go to durable storage first, then sync back to S3 automatically
- Unused data expires on a configurable window (1-365 days, default 30), keeping the hot tier lean
- Up to 25,000 concurrent connections across any AWS compute (EC2, ECS, Lambda, SageMaker)
- Dual access - the same bucket is simultaneously accessible via file system and standard S3 API
Best suited for
- ML training pipelines and data preparation that expect a POSIX file system
- AI agent coordination with shared workspaces across many compute nodes
- Scientific and bioinformatics workflows using legacy file-based tools
- Team collaboration on large shared datasets without duplicating data into EFS
# Pre-requisite: create a File System access point in the S3 Console. # The NFS hostname is displayed in the console once the access point is ready. # ---- EC2 / ECS host: mount via NFS4 ---- sudo mount -t nfs4 \ -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600 \ <nfs-hostname-from-console>.s3-accesspoint.us-east-1.amazonaws.com:/ \ /mnt/s3files # Standard POSIX operations - no application code changes needed ls /mnt/s3files/training-data/ cp ./model_v2.bin /mnt/s3files/models/ cat /mnt/s3files/config/settings.json # Objects written via NFS are immediately accessible via the S3 API aws s3 ls s3://my-ml-bucket/models/
S3 Files: Dual-Access Architecture
Compute nodes mount the bucket as a POSIX file system; existing S3 tooling continues to work unchanged via the object API.
When not to use it: if your pipeline already uses the S3 API naturally, stick with it. S3 Files targets workloads that today copy data out of S3 into EFS or FSx just to get file semantics. Pay-as-you-go with no provisioned capacity required.
7. S3 Storage Lens
Storage Lens is an analytics dashboard that aggregates storage metrics across every bucket in your account, or across all accounts in an AWS Organization. Instead of clicking into each bucket individually, you get a unified view of usage trends, activity rates, and cost-optimization recommendations in one place.
| Feature | Free Tier | Advanced Tier |
|---|---|---|
| Cost | Free | ~$0.20 / million objects / month |
| Metrics | 28 usage metrics | 35+ including activity metrics |
| Scope | Single account | Multi-account / Organization |
| Recommendations | Basic | Detailed cost-optimization suggestions |
| CloudWatch publish | No | Yes |
| Prefix-level breakdown | No | Yes (configurable depth) |
The most actionable metrics for cost control are:
Non-current version storage
Bytes held by old object versions when versioning is enabled. Usually safe to expire after a retention window.
Incomplete multipart uploads
Parts from aborted uploads that accumulate silently. Add a lifecycle rule to abort incomplete MPUs after 7 days.
Objects not accessed in 90+ days
Storage Lens surfaces these buckets as Intelligent-Tiering candidates, saving up to 40% on storage cost.
# Create a Storage Lens dashboard with Advanced metrics enabled
aws s3control put-storage-lens-configuration \
--account-id 123456789012 \
--config-id my-org-lens \
--storage-lens-configuration file://lens-config.json
# lens-config.json
{
"Id": "my-org-lens",
"IsEnabled": true,
"AccountLevel": {
"ActivityMetrics": { "IsEnabled": true },
"BucketLevel": {
"ActivityMetrics": { "IsEnabled": true },
"PrefixLevel": {
"StorageMetrics": {
"IsEnabled": true,
"SelectionCriteria": { "MaxDepth": 2, "MinStorageBytes": 1048576 }
}
}
}
},
"DataExport": {
"S3BucketDestination": {
"Format": "Parquet",
"OutputSchemaVersion": "V_1",
"AccountId": "123456789012",
"Arn": "arn:aws:s3:::my-lens-exports"
}
}
}Default dashboard: Storage Lens is enabled automatically at the account level with 28 free usage metrics. Check it in the S3 console under Storage Lens → Dashboards. Enable the Advanced tier only if you need per-prefix activity breakdowns or cross-account aggregation.
8. Modern S3 Capabilities
Over the past years AWS has extended S3 beyond raw object storage into structured data, vector search, and automatic metadata cataloging. These three capabilities reduce the need for separate purpose-built services in many common workflows.
8a. S3 Tables
S3 Tables is a new bucket type that stores data in Apache Iceberg format and manages it for you. AWS runs compaction, snapshot cleanup, and concurrent write coordination automatically - no Glue catalog setup, no Spark vacuum jobs, no manual partition management.
- What it stores: structured tabular data as Parquet files with Iceberg metadata
- AWS manages: automatic compaction, snapshot expiry, concurrent write coordination (optimistic concurrency)
- Query engines: Athena, EMR (Spark / Hive), Redshift Spectrum - all read natively with no extra setup
- vs DIY Iceberg on S3: eliminates compaction jobs, Glue catalog maintenance, and vacuum scripts
- Best for: data lake tables that multiple analytics engines need to query and write concurrently
import boto3
# Create an S3 Tables bucket - a specialized bucket for structured tabular data
tables = boto3.client("s3tables", region_name="us-east-1")
bucket = tables.create_table_bucket(name="analytics-tables")
bucket_arn = bucket["arn"]
# Create a namespace (logical grouping, similar to a database schema)
tables.create_namespace(tableBucketARN=bucket_arn, namespace=["sales"])
# Create an Iceberg table - AWS handles compaction and snapshots automatically
tables.create_table(
tableBucketARN=bucket_arn,
namespace="sales",
name="orders",
format="ICEBERG",
)
# Query directly from Athena - no Glue catalog setup required
# (catalog format: "s3tablescatalog/<bucket-name>"."<namespace>"."<table>")
# SELECT customer_id, SUM(total_amount) AS revenue
# FROM "s3tablescatalog/analytics-tables"."sales"."orders"
# WHERE order_date >= DATE '2026-01-01'
# GROUP BY customer_id
# ORDER BY revenue DESC;8b. S3 Vectors
S3 Vectors (preview, 2025) adds a vector index directly to S3. Store high-dimensional embeddings alongside your objects and run approximate nearest-neighbor (ANN) search without spinning up a separate vector database.
- How it works: create an index (specifying dimension and distance metric), then put vectors with metadata; query returns the top-K most similar vectors
- Distance metrics: cosine similarity, euclidean distance, dot product
- Use cases: semantic document search, image similarity, RAG pipelines, recommendation systems
- Replaces: a separate Pinecone, OpenSearch k-NN, or pgvector deployment for many workloads
- Limits: suited for hundreds of millions of vectors; for sub-10 ms latency at extreme scale, a purpose-built vector DB still wins
import boto3
client = boto3.client("s3vectors", region_name="us-east-1")
# Create a vector index - dimension must match your embedding model
client.create_index(
vectorBucketName="my-vectors-bucket",
indexName="doc-embeddings",
dataType="float32",
dimension=1536, # matches OpenAI text-embedding-3-small
distanceMetric="cosine",
)
# Store a vector alongside metadata
client.put_vectors(
vectorBucketName="my-vectors-bucket",
indexName="doc-embeddings",
vectors=[{
"key": "reports/q1-2026.pdf",
"data": {"float32": embedding_array}, # your 1536-float list
"metadata": {"department": "finance", "year": "2026"}
}]
)
# Nearest-neighbor search - no separate vector DB required
results = client.query_vectors(
vectorBucketName="my-vectors-bucket",
indexName="doc-embeddings",
queryVector={"float32": query_embedding},
topK=5,
)
for match in results["vectors"]:
print(match["key"], round(match["score"], 4), match["metadata"])8c. S3 Metadata
S3 Metadata (2025) automatically extracts and indexes object metadata as objects land in a bucket. The index is stored in an S3 Tables bucket and is queryable with Athena using SQL - no object downloads or bucket scans required.
- What gets indexed: system metadata (size, ETag, last-modified, content-type) plus all user-defined tags
- Real-time: metadata is indexed as objects are written; unlike S3 Inventory (daily batch), queries reflect the current state
- Zero pipeline changes: enable it on an existing bucket and S3 backfills metadata for all existing objects
- Built on S3 Tables: the metadata table is an Iceberg table you can join against your own data tables
- Example query: find all PDFs tagged
department=financeuploaded in the last 7 days without touching any object
import boto3
# Step 1: create a metadata table bucket once per region (uses S3 Tables under the hood)
tables = boto3.client("s3tables", region_name="us-east-1")
meta_bucket = tables.create_table_bucket(name="s3-metadata-store")
# Step 2: enable automatic metadata indexing on your data bucket
s3 = boto3.client("s3")
s3.put_bucket_metadata_table_configuration(
Bucket="my-app-bucket-2026",
MetadataTableConfiguration={
"S3TablesDestination": {
"TableBucketARN": meta_bucket["arn"],
"TableName": "my-app-bucket-2026"
}
}
)
# S3 now indexes every object as it lands - no pipeline changes needed
# Query metadata via Athena: find recent PDFs tagged department=finance
# (verify the exact namespace in the S3 Tables console after enabling metadata)
# SELECT key, size, last_modified
# FROM "s3tablescatalog/s3-metadata-store"."aws_s3_metadata"."my-app-bucket-2026"
# WHERE content_type = 'application/pdf'
# AND json_extract_scalar(user_metadata, '$.department') = 'finance'
# AND last_modified >= NOW() - INTERVAL '7' DAY;Key Takeaways
- Buckets are globally unique containers; objects inside have no real directory structure, just key prefixes
- Storage classes trade storage cost for retrieval speed/cost; use INTELLIGENT_TIERING when access patterns are unknown
- Versioning protects against accidental deletes and overwrites; use with lifecycle rules to expire old versions
- Lifecycle policies automatically tier or expire data, saving significant costs for log archives
- Presigned URLs grant time-limited access to private objects; use for user downloads and browser-side uploads
- Static hosting - pair S3 with CloudFront for HTTPS, caching, and a custom domain on static sites
- S3 Files mounts a bucket as an NFS file system; eliminates data duplication for ML and file-based workloads that need POSIX access
- Storage Lens provides org-wide metrics and cost-saving recommendations; free tier covers basic usage and is enabled by default
- S3 Tables adds native Apache Iceberg table support with automatic compaction; query with Athena, EMR, or Redshift without managing a Glue catalog
- S3 Vectors stores and queries vector embeddings directly in S3 using ANN search; reduces need for a separate vector database in RAG and semantic search workloads
- S3 Metadata automatically indexes every object as it lands and exposes a live SQL-queryable metadata table; replaces manual bucket scanning for object discovery