AWS Lake Formation
Build, secure, and manage data lakes in days, not months
Why Lake Formation?
Setting up a data lake manually involves configuring complicated S3 bucket policies, IAM roles, and Glue permissions across multiple accounts. It's a "Data Swamp" of security risks. AWS Lake Formation solves this by providing a single, centralized layer to define security, governance, and auditing policies. You define permissions once ("Data Scientists can read Sales data"), and it enforces them across Athena, Redshift, and Glue.
Core Concepts
🔐 Centralized Permissions
Manage access to databases and tables in one place.
- Grant/Revoke access
- Column-level security
- Row-level filtering
📋 Blueprints
Templated data ingestion workflows.
- Ingest from Database
- Ingest from Log Files
- Incremental loads
🛡️ Governance
Audit and control data access.
- Audit logs (who accessed what)
- Data Catalog management
- LF-Tags (Tag-based access)
Step 1: Register Data Lake Location
Tell Lake Formation where your data lives (S3)
Before Lake Formation can manage permissions, you must "register" your S3 buckets. This hands over access control delegation to the Lake Formation service.
Register Resource
Action: Register S3 Path
Prerequisite: IAM Role with S3 Access
Impact: Lake Formation Permissions take over
import boto3
client = boto3.client('lakeformation', region_name='us-east-1')
# Register an S3 bucket as a data lake location
response = client.register_resource(
ResourceArn='arn:aws:s3:::my-corporate-data-lake',
UseServiceLinkedRole=True # Use default service role
)
print(f"Resource Registered: {response['ResponseMetadata']['HTTPStatusCode']}")No one can access data in this bucket via Glue/Athena unless explicitly granted permission here.
Don't lock yourself out! Use "Hybrid Access Mode" during migration. This lets you use both Lake Formation permissions AND existing IAM/S3 policies simultaneously until you are ready to switch over completely.
Step 2: Grant Permissions
Fine-grained access control (Column/Row level)
Unlike IAM (which is bucket-level), Lake Formation allows you to grant access to specific Tables and even specific Columns.
Grant Column-Level Access
# Grant SELECT permission on specific columns to a User
response = client.grant_permissions(
Principal={
'DataLakePrincipalIdentifier': 'arn:aws:iam::123456789012:user/data-scientist-alice'
},
Resource={
'Table': {
'DatabaseName': 'sales_db',
'Name': 'transactions',
'ColumnNames': ['date', 'amount', 'product_id'] # Exclude PII like 'customer_email'
}
},
Permissions=['SELECT'],
PermissionsWithGrantOption=[]
)
print("Permissions Granted: User can ONLY see non-PII columns.")SELECT * FROM transactions, she will ONLY see the 3 allowed columns.Attempting to query
customer_email results in an Access Denied error automatically.Step 2.5: Row-Level Security (Data Filters)
Restrict specific rows based on SQL-like filters
How it works
Create a Data Filter that applies a WHERE clause (e.g., country_code='US') automatically. When the user queries the table, they only see rows matching that filter.
# Create a 'Filter' that only shows 'US' region data
client.create_data_cells_filter(
TableData={
'DatabaseName': 'sales_db',
'TableName': 'transactions',
'Name': 'US_Only_Filter',
'RowFilter': { 'FilterExpression': "country_code = 'US'" },
'ColumnNames': ['date', 'amount', 'product_id'] # Can also restrict columns!
}
)
# Grant the filter to the US Analyst
client.grant_permissions(
Principal={'DataLakePrincipalIdentifier': 'arn:aws:iam::...:user/us_analyst'},
Resource={'DataCellsFilter': {
'DatabaseName': 'sales_db', 'TableName': 'transactions', 'Name': 'US_Only_Filter'
}},
Permissions=['SELECT']
)Step 3: Scale with LF-Tags
Manage thousands of tables easily with Tags
Granting permission table-by-table is tedious. Instead, assign Tags to databases and tables (e.g., CostCenter=Finance or DataClass=Sensitive) and grant permissions on the Tag.
1. Assign Tag to Table
transactions table is tagged Class=Restricted
2. Grant on Tag
Grant SELECT on all resources where Class=Restricted
# Efficiently grant access using Tags
response = client.grant_permissions(
Principal={
'DataLakePrincipalIdentifier': 'arn:aws:iam::123456789012:role/ComplianceRole'
},
Resource={
'LFTagPolicy': {
'ResourceType': 'TABLE',
'Expression': [
{
'TagKey': 'Classification',
'TagValues': ['Public', 'Internal'] # Allow Access
}
# Implicitly denies 'Confidential'
]
}
},
Permissions=['DESCRIBE', 'SELECT']
)No need to update permissions when adding new datasets!
Step 4: Cross-Account Sharing
Zero-copy sharing across AWS Accounts
Powered by AWS RAM
Lake Formation automates AWS Resource Access Manager (RAM). When you grant permissions to an external Account ID, the table appears in their Glue Catalog as a "Resource Link". They can query it immediately without copying a single byte of data!
Production Readiness: Common Pitfalls
By default, new tables allow "IAMAllowedPrincipals". You MUST revoke this permission, otherwise, Lake Formation rules are ignored and standard IAM policies still apply (allowing full access).
If your S3 bucket is encrypted, Lake Formation needs kms:Decrypt permissions on the key. Without this, Athena queries and previews will fail with obscure errors.
Shared tables don't magically appear. In the consumer account, you must explicitly create a Resource Link (symlink) in your Glue Data Catalog to query the shared data.
When to use AWS Lake Formation?
✅ Ideal Scenarios
- Multi-tenant Data Lakes: Different teams (Marketing, Sales) accessing shared S3 bucket.
- PII Protection: Need to hide specific columns (SSN, Email) from analysts.
- Audit Compliance: Need a central log of "Who queried this table?"
- Cross-Account Access: Share data securely between AWS accounts.
❌ When to Skip
- Small Projects: Single bucket, single team? S3 Policies are simpler.
- Non-Tabular Data: Lake Formation excels at Database/Table structures, less so for raw unstructured files.
- Real-time Streaming Latency: Lake Formation adds a tiny overhead layer.