Secrets Management & Key Rotation

dotenv, HashiCorp Vault, AWS Secrets Manager, SSM Parameter Store, Key Rotation

Introduction

Hardcoded secrets in source code are one of the most common causes of security breaches. GitHub scans show that hundreds of thousands of API keys, database passwords, and private keys are accidentally committed every year. This lesson covers a layered approach: environment variables for development, secrets managers for production, and key rotation strategies to limit the blast radius when a secret is eventually compromised.

Why Environment Variables Alone Are Not Enough

Problems with Env Vars
  • Any process on the machine can read env vars of its parent
  • Env vars are logged in crash reports, CI/CD logs, and error trackers
  • No audit trail, you can't see who accessed the secret or when
  • No automatic rotation, manual updates are error-prone and often skipped
  • Secrets often end up in .env files accidentally committed to git
Secrets Manager Benefits
  • Centralized storage with fine-grained access control
  • Full audit log of every secret access
  • Automatic rotation with zero-downtime workflows
  • Versioning, keep old versions during rotation
  • Dynamic secrets, generate per-application credentials

python-dotenv + Startup Validation

pydantic-settings extends Pydantic v2 to handle application configuration. Its BaseSettings class automatically reads values from environment variables, .env files, and other sources in a defined priority order. The key features for secrets management are:

Key Concepts
  • BaseSettings - auto-reads from env vars and .env files; env vars take priority over the file
  • SettingsConfigDict(env_file=".env") - Pydantic v2 way to configure the settings source
  • SecretStr - wraps a string so it never appears in repr(), logs, or stack traces; use .get_secret_value() only when the raw value is actually needed
  • Field(...) - the ... marks a field as required; missing env vars raise a ValidationError at startup
Why Fail Fast?
  • Discovering a missing secret at startup is far better than discovering it mid-request when a user is waiting
  • A clear ValidationError: jwt_secret_key field required is easier to debug than a KeyError buried in application logic
  • In production, follow the startup error with sys.exit(1) so the container or process supervisor restarts with an alert rather than silently degrading

Install: pip install python-dotenv pydantic-settings

# config.py, Secrets validation at startup with python-dotenv + pydantic
import os
from pydantic import Field, SecretStr
from pydantic_settings import BaseSettings, SettingsConfigDict


class Settings(BaseSettings):
    """Application configuration, validated at startup."""

    model_config = SettingsConfigDict(
        env_file=".env",
        env_file_encoding="utf-8",
    )

    # Database
    database_url: SecretStr = Field(..., description="PostgreSQL connection string")
    database_pool_size: int = Field(default=5, ge=1, le=50)

    # Authentication
    jwt_secret_key: SecretStr = Field(..., min_length=32)
    jwt_algorithm: str = Field(default="HS256")

    # External APIs
    stripe_api_key: SecretStr | None = Field(default=None)

    # App
    debug: bool = Field(default=False)
    environment: str = Field(default="production")

    def validate_production_settings(self) -> None:
        """Extra validation for production environments."""
        if self.environment == "production":
            assert not self.debug, "DEBUG must be False in production"
            assert self.stripe_api_key is not None, "Stripe API key required in production"

# Load and validate at startup, fail fast if secrets are missing
try:
    settings = Settings()
    settings.validate_production_settings()
    print("✅ All secrets loaded and validated")
    print(f"   Environment: {settings.environment}")
    print(f"   JWT Algorithm: {settings.jwt_algorithm}")
    # Never print secret values!
    # settings.jwt_secret_key.get_secret_value(), only access when needed
except Exception as e:
    print(f"❌ Configuration error: {e}")
    # In production: sys.exit(1), fail fast rather than running with invalid config
Expected Output:
✅ All secrets loaded and validated
   Environment: development
   JWT Algorithm: HS256

HashiCorp Vault, Python hvac Client

HashiCorp Vault is an open-source tool for securely storing and accessing secrets. It provides a unified interface to any secret while providing tight access control and recording a detailed audit log. Vault solves the problem of secret sprawl: instead of credentials scattered across config files, environment variables, and code repositories, all secrets live in one encrypted, auditable location. Vault supports multiple secret engines (key/value, database dynamic credentials, PKI certificates, AWS/GCP IAM) and authentication methods (token, AppRole, Kubernetes, AWS IAM).

Prerequisite: Start a Local Vault Dev Server

The examples below connect to localhost:8200. Start a local dev server first (requires the Vault CLI):

vault server -dev -dev-root-token-id="dev-token"

The dev server starts unsealed, stores data in memory, and automatically sets VAULT_ADDR=http://127.0.0.1:8200. It is not suitable for production use.

Install: pip install hvac

# vault_client.py
# Requires: vault server -dev (for testing)
import hvac
import os

VAULT_URL = os.getenv("VAULT_ADDR", "http://localhost:8200")
VAULT_TOKEN = os.getenv("VAULT_TOKEN", "dev-token")   # Use AppRole/AWS auth in production

def get_vault_client() -> hvac.Client:
    """Create an authenticated Vault client."""
    client = hvac.Client(url=VAULT_URL, token=VAULT_TOKEN)
    assert client.is_authenticated(), "Vault authentication failed"
    return client

client = get_vault_client()

# ── Write secrets to KV v2 store ──────────────────────────────
client.secrets.kv.v2.create_or_update_secret(
    path="myapp/database",
    secret={
        "url": "postgresql://user:pass@db:5432/mydb",
        "pool_size": "10",
    },
)
print("✅ Secret written to vault: myapp/database")

# ── Read secrets ─────────────────────────────────────────────
secret = client.secrets.kv.v2.read_secret_version(path="myapp/database")
data = secret["data"]["data"]
print(f"Database URL: {data['url'][:30]}...")   # Never log full secret

# ── Dynamic secrets, Vault generates DB credentials ─────────
# (Requires database secrets engine configured)
# lease = client.secrets.database.generate_credentials(name="myapp-role")
# temp_username = lease["data"]["username"]   # Expires automatically!
# temp_password = lease["data"]["password"]

# ── Rotate a secret ──────────────────────────────────────────
client.secrets.kv.v2.create_or_update_secret(
    path="myapp/database",
    secret={"url": "postgresql://newuser:newpass@db:5432/mydb", "pool_size": "10"},
)

# KV v2 keeps history, you can access previous versions
prev = client.secrets.kv.v2.read_secret_version(path="myapp/database", version=1)
print(f"Previous version retained: {prev['data']['metadata']['version']}")

AWS Secrets Manager

AWS Secrets Manager is a fully managed service for storing, rotating, and retrieving credentials, API keys, and other secrets. All secrets are encrypted at rest using AWS KMS and in transit via TLS. Access is controlled by IAM policies, an EC2 instance or Lambda function retrieves a secret via its IAM role with no hard-coded credentials needed.

Its standout feature is automatic rotation: Secrets Manager invokes an AWS Lambda function on a schedule to generate new credentials, update the target service (e.g. RDS), and atomically swap to the new secret, all with zero downtime. It natively integrates with RDS, Redshift, and DocumentDB for one-click rotation. Trade-off: approximately $0.40 per secret per month + $0.05 per 10,000 API calls. For plain configuration values that do not need rotation, SSM Parameter Store (covered below) is a cheaper alternative.

Install: pip install boto3

# aws_secrets.py
import boto3
import json
from functools import lru_cache
from datetime import datetime, timezone, timedelta

# ── Retrieve a secret ─────────────────────────────────────────
def get_secret(secret_name: str, region_name: str = "us-east-1") -> dict:
    """Retrieve a secret from AWS Secrets Manager."""
    client = boto3.client("secretsmanager", region_name=region_name)
    response = client.get_secret_value(SecretId=secret_name)

    if "SecretString" in response:
        return json.loads(response["SecretString"])
    else:
        # Binary secret
        return {"binary": response["SecretBinary"]}

# ── Cache with TTL, don't call AWS on every request ─────────
_secret_cache: dict[str, tuple[dict, datetime]] = {}
CACHE_TTL = timedelta(minutes=5)

def get_secret_cached(secret_name: str) -> dict:
    """Get secret with 5-minute cache to reduce AWS API calls."""
    cached = _secret_cache.get(secret_name)
    if cached:
        value, fetched_at = cached
        if datetime.now(timezone.utc) - fetched_at < CACHE_TTL:
            return value

    value = get_secret(secret_name)
    _secret_cache[secret_name] = (value, datetime.now(timezone.utc))
    return value

# ── Key rotation pattern ──────────────────────────────────────
def rotate_database_secret(secret_name: str, new_credentials: dict) -> None:
    """
    Rotate a database secret with zero downtime.
    Steps:
    1. Create new credentials in the database
    2. Update the secret with new + old credentials (AWSPENDING)
    3. Test new credentials
    4. Finalize rotation (AWSCURRENT = new)
    """
    client = boto3.client("secretsmanager", region_name="us-east-1")

    # AWS Secrets Manager can automate this with Lambda rotation functions
    client.rotate_secret(
        SecretId=secret_name,
        RotationRules={"AutomaticallyAfterDays": 30},
        RotateImmediately=True,
    )

print("AWS Secrets Manager patterns:")
print("  • get_secret(), retrieve and parse JSON secret")
print("  • Cache with TTL, avoid API throttling (10,000 req/month free tier)")
print("  • Automatic rotation, Lambda function updates DB credentials")
print("  • IAM roles, EC2/Lambda instances access secrets via role, no credentials needed")

AWS SSM Parameter Store

AWS Systems Manager (SSM) Parameter Store is a general-purpose configuration and secrets store. Unlike Secrets Manager, it is designed for both plain configuration values (Standard type, free up to 10,000 parameters) and encrypted secrets (SecureString type, encrypted via AWS KMS). It does not support automatic rotation natively, but it is significantly cheaper, standard parameters are free and advanced parameters cost $0.05 per parameter per month.

Use SSM Parameter Store when…
  • You need to store many configuration values (feature flags, URLs, non-sensitive config)
  • Cost is a concern and automatic rotation is not required
  • You want hierarchical namespacing (/myapp/prod/db_url)
  • You need to share config across multiple Lambda functions or EC2 instances
Use Secrets Manager when…
  • You need automatic rotation (database passwords, API keys)
  • You need cross-account secret sharing
  • You want native RDS / Redshift / DocumentDB integration
  • Fine-grained per-secret resource-based policies are required

Install: pip install boto3

# ssm_parameters.py
import boto3
import json

ssm = boto3.client("ssm", region_name="us-east-1")

# ── Write a plain String parameter ────────────────────────────
ssm.put_parameter(
    Name="/myapp/prod/log_level",
    Value="INFO",
    Type="String",
    Overwrite=True,
)

# ── Write an encrypted SecureString parameter ─────────────────
ssm.put_parameter(
    Name="/myapp/prod/db_password",
    Value="s3cr3t-db-pass",
    Type="SecureString",   # Encrypted with AWS KMS
    Overwrite=True,
)

# ── Read a single parameter (decrypted automatically) ─────────
response = ssm.get_parameter(
    Name="/myapp/prod/db_password",
    WithDecryption=True,   # Required for SecureString
)
print(f"DB password: {'*' * len(response['Parameter']['Value'])}")

# ── Read all parameters under a path (hierarchical) ───────────
paginator = ssm.get_paginator("get_parameters_by_path")
for page in paginator.paginate(
    Path="/myapp/prod/",
    WithDecryption=True,
    Recursive=True,
):
    for param in page["Parameters"]:
        name = param["Name"].split("/")[-1]
        ptype = param["Type"]
        print(f"  {name} ({ptype}): {'[encrypted]' if ptype == 'SecureString' else param['Value']}")

# ── Delete a parameter ────────────────────────────────────────
# ssm.delete_parameter(Name="/myapp/prod/log_level")
Expected Output:
DB password: **************
  log_level (String): INFO
  db_password (SecureString): [encrypted]

Key Takeaways

  • Never hardcode secrets, no API keys, passwords, or private keys in source code; use pre-commit hooks to prevent it
  • Validate all secrets at startup, fail fast with clear error messages rather than discovering missing secrets at runtime
  • Use Pydantic SecretStr, prevents secrets from appearing in logs, repr(), or stack traces
  • Vault and AWS SM provide audit trails, every access is logged with who accessed what and when
  • SSM Parameter Store vs Secrets Manager, use SSM for cheap config storage; use Secrets Manager when automatic rotation or cross-account sharing is needed
  • Cache secrets with TTL, don't call secrets managers on every request; 5-minute cache is a good default
  • Rotate secrets regularly, limit breach impact; automate rotation to remove the operational burden
What's Next?

Lesson 16 covers security testing, how to find vulnerabilities before attackers do, using automated tools and penetration testing methodology.

  • bandit, Python SAST for finding common vulnerabilities
  • pip audit, scanning dependencies for known CVEs
  • Fuzzing with Hypothesis, property-based testing for security
  • Penetration testing methodology, PTES phases