Architecture Overview
Diagram powered by Lucidchart
Ingestion Layer
Captures data from all sources in real time (Kinesis, Firehose) and in batch (DMS/CDC, S3 Transfer).
Bronze - Raw
Stores data exactly as received from the source, with no transformation. Acts as the system of record.
Silver - Curated
Cleaned, validated, and normalized data. Glue ETL applies business rules; the Crawler keeps the Data Catalog up to date.
Gold - Analytics-Ready
Aggregated and query-optimized tables loaded into Redshift, queryable via Athena, or processed by EMR for ML features.
Consumption
Dashboards in QuickSight, data product APIs via API Gateway, and model training pipelines in SageMaker.