Configuration Management

Master configuration management with INI, YAML, TOML files, environment variables, and the python-dotenv library for professional application settings.

Why Configuration Management?

Hardcoding configuration values makes applications inflexible and insecure. Configuration management separates settings from code, enabling different configurations for development, testing, and production without code changes.

Why Configuration Management Matters

Benefits

Separation of concerns: Code vs. configuration
Environment flexibility: Different settings per environment
Security: Keep secrets out of version control
Easy deployment: Change settings without redeploying
Team collaboration: Developers use their own configs

Problems with Hardcoding

Secrets exposed in code repositories
Can't switch between dev/prod databases
Code changes required for config updates
Difficult to test with different settings
No way to override settings at runtime

Configuration Hierarchy

A common pattern is to layer configuration sources, with each layer overriding the previous:

1. Default values (in code)
2. Configuration file (config.ini, config.yaml)
3. Environment variables (.env, system env)
4. Command-line arguments

Later sources override earlier ones

Environment Variables

Environment variables are key-value pairs set at the operating system level. They're perfect for configuration that changes between environments or contains sensitive data.

Reading Environment Variables

import os

# Get environment variable (raises KeyError if not found)
# database_url = os.environ['DATABASE_URL']  # Dangerous!

# Get with default value (safer)
database_url = os.environ.get('DATABASE_URL', 'sqlite:///default.db')
print(f"Database: {database_url}")
# Database: sqlite:///default.db

# Check if variable exists
if 'API_KEY' in os.environ:
    api_key = os.environ['API_KEY']
    print("API key found")
else:
    print("API key not set")

Type Conversion for Environment Variables

Environment variables are always strings, so you need to convert them to the appropriate type.

import os

# Boolean values
debug = os.environ.get('DEBUG', 'False').lower() in ('true', '1', 'yes')
print(f"Debug mode: {debug}")
# Debug mode: False

# Integer values
max_connections = int(os.environ.get('MAX_CONNECTIONS', '10'))
print(f"Max connections: {max_connections}")
# Max connections: 10

# Float values
timeout = float(os.environ.get('TIMEOUT', '30.0'))
print(f"Timeout: {timeout}s")
# Timeout: 30.0s

# List values (comma-separated)
allowed_hosts = os.environ.get('ALLOWED_HOSTS', 'localhost').split(',')
print(f"Allowed hosts: {allowed_hosts}")
# Allowed hosts: ['localhost']

Security Note: Never commit sensitive environment variables to version control. Use .env files (and add them to .gitignore) or a secrets management service.

.env Files and python-dotenv

.env files store environment variables in a simple format. The python-dotenvlibrary loads these files into your application.

Install python-dotenv: pip install python-dotenv

Creating a .env File

# .env file

# Database configuration
DATABASE_URL=postgresql://user:pass@localhost/mydb
DATABASE_POOL_SIZE=10

# API keys
API_KEY=your-secret-api-key-here
SECRET_KEY=super-secret-key-12345

# Application settings
DEBUG=True
LOG_LEVEL=INFO
ALLOWED_HOSTS=localhost,127.0.0.1

Loading .env Files

# app.py
import os
from dotenv import load_dotenv

# Load .env file
load_dotenv()

# Now environment variables from .env are available
database_url = os.environ.get('DATABASE_URL')
api_key = os.environ.get('API_KEY')
debug = os.environ.get('DEBUG', 'False').lower() == 'true'

print(f"Database: {database_url}")
print(f"Debug mode: {debug}")

# Output:
# Database: postgresql://user:pass@localhost/mydb
# Debug mode: True

Best Practices for .env Files

DO

Add .env to .gitignore
Create .env.example with dummy values
Document required variables in README
Use descriptive variable names
Keep secrets only in .env, not in code

DON'T

Commit .env files to version control
Share .env files via email or chat
Use production secrets in development
Store .env in publicly accessible directories
Forget to update .env.example when adding variables

# .env.example - Template for .env file

DATABASE_URL=postgresql://user:password@localhost/dbname
API_KEY=your-api-key-here
SECRET_KEY=your-secret-key-here
DEBUG=False
LOG_LEVEL=INFO

INI Files with configparser

INI files are a classic configuration format with sections and key-value pairs. Python'sconfigparser module handles them easily.

INI File Format

# config.ini

[database]
host = localhost
port = 5432
name = myapp
user = admin
password = secret123

[api]
base_url = https://api.example.com
timeout = 30
max_retries = 3

[app]
debug = true
log_level = INFO
max_workers = 4

Reading INI Files

import configparser

# Create parser
config = configparser.ConfigParser()

# Read config file
config.read('config.ini')

# Access values (always returns strings)
db_host = config['database']['host']
db_port = config['database']['port']
db_name = config['database']['name']

print(f"Database: {db_name} on {db_host}:{db_port}")
# Database: myapp on localhost:5432

# Access API settings
api_url = config['api']['base_url']
timeout = config['api']['timeout']

print(f"API URL: {api_url}")
print(f"Timeout: {timeout}")
# API URL: https://api.example.com
# Timeout: 30

Type Conversion in configparser

import configparser

config = configparser.ConfigParser()
config.read('config.ini')

# Get as integer
db_port = config.getint('database', 'port')
max_workers = config.getint('app', 'max_workers')
print(f"Port: {db_port} (type: {type(db_port).__name__})")
# Port: 5432 (type: int)

# Get as float
timeout = config.getfloat('api', 'timeout')
print(f"Timeout: {timeout} (type: {type(timeout).__name__})")
# Timeout: 30.0 (type: float)

# Get as boolean
debug = config.getboolean('app', 'debug')
print(f"Debug: {debug} (type: {type(debug).__name__})")
# Debug: True (type: bool)

# Get with fallback default
api_key = config.get('api', 'api_key', fallback='default-key')
print(f"API Key: {api_key}")
# API Key: default-key (not in config, so uses fallback)

# Check if section or option exists
if config.has_section('database'):
    print("Database section exists")

if config.has_option('database', 'host'):
    print("Database host is configured")

# List all sections
print(f"Sections: {config.sections()}")
# Sections: ['database', 'api', 'app']

Writing INI Files

import configparser

# Create new config
config = configparser.ConfigParser()

# Add sections and values
config['database'] = {
    'host': 'localhost',
    'port': '5432',
    'name': 'myapp'
}

config['app'] = {}
config['app']['debug'] = 'False'
config['app']['log_level'] = 'WARNING'

# Write to file
with open('new_config.ini', 'w') as configfile:
    config.write(configfile)

print("Configuration file created")

YAML Configuration Files

YAML (YAML Ain't Markup Language) is a human-friendly data serialization format. It's popular for configuration because it's clean and supports complex nested structures.

Install PyYAML: pip install pyyaml

YAML File Format

# config.yaml

database:
  host: localhost
  port: 5432
  name: myapp
  credentials:
    user: admin
    password: secret123
  pool:
    min_size: 5
    max_size: 20

api:
  base_url: https://api.example.com
  timeout: 30.0
  endpoints:
    - /users
    - /posts
    - /comments

app:
  debug: true
  log_level: INFO
  features:
    auth: true
    cache: false

Reading YAML Files

import yaml

# Read YAML file
with open('config.yaml', 'r') as file:
    config = yaml.safe_load(file)

# Access nested values
db_host = config['database']['host']
db_port = config['database']['port']
db_user = config['database']['credentials']['user']

print(f"Database: {db_user}@{db_host}:{db_port}")
# Database: admin@localhost:5432

# Access lists
endpoints = config['api']['endpoints']
print(f"API Endpoints: {endpoints}")
# API Endpoints: ['/users', '/posts', '/comments']

# Types are automatically converted
timeout = config['api']['timeout']
debug = config['app']['debug']
print(f"Timeout: {timeout} (type: {type(timeout).__name__})")
print(f"Debug: {debug} (type: {type(debug).__name__})")
# Timeout: 30.0 (type: float)
# Debug: True (type: bool)

Writing YAML Files

import yaml

# Create configuration dictionary
config = {
    'database': {
        'host': 'localhost',
        'port': 5432,
        'credentials': {
            'user': 'admin',
            'password': 'secret'
        }
    },
    'app': {
        'debug': False,
        'features': ['auth', 'cache', 'api']
    }
}

# Write to YAML file
with open('output.yaml', 'w') as file:
    yaml.dump(config, file, default_flow_style=False)

print("YAML configuration written")

Security: Always use yaml.safe_load() instead of yaml.load()to prevent arbitrary code execution from untrusted YAML files.

TOML Configuration Files

TOML (Tom's Obvious, Minimal Language) is designed to be easy to read and write. Python 3.11+ includes built-in TOML support via tomllib.

Python 3.11+: Use built-in tomllib
Python < 3.11: Install pip install tomli (read) or pip install tomli-w (write)

TOML File Format

# config.toml

[database]
host = "localhost"
port = 5432
name = "myapp"

[database.credentials]
user = "admin"
password = "secret123"

[database.pool]
min_size = 5
max_size = 20

[api]
base_url = "https://api.example.com"
timeout = 30.0
endpoints = ["/users", "/posts", "/comments"]

[app]
debug = true
log_level = "INFO"

[app.features]
auth = true
cache = false
api = true

Reading TOML Files

# For Python 3.11+
import tomllib

# For Python < 3.11, use: import tomli as tomllib

# Read TOML file (must open in binary mode)
with open('config.toml', 'rb') as file:
    config = tomllib.load(file)

# Access values
db_host = config['database']['host']
db_port = config['database']['port']
db_user = config['database']['credentials']['user']

print(f"Database: {db_user}@{db_host}:{db_port}")
# Database: admin@localhost:5432

# Types are automatically converted
timeout = config['api']['timeout']
debug = config['app']['debug']
endpoints = config['api']['endpoints']

print(f"Timeout: {timeout} (type: {type(timeout).__name__})")
print(f"Debug: {debug} (type: {type(debug).__name__})")
print(f"Endpoints: {endpoints}")
# Timeout: 30.0 (type: float)
# Debug: True (type: bool)
# Endpoints: ['/users', '/posts', '/comments']

Writing TOML Files

# For writing TOML, use tomli-w (or tomllib in Python 3.11+ doesn't support writing)
# pip install tomli-w

import tomli_w

config = {
    'database': {
        'host': 'localhost',
        'port': 5432,
        'credentials': {
            'user': 'admin',
            'password': 'secret'
        }
    },
    'app': {
        'debug': False,
        'log_level': 'INFO'
    }
}

# Write to TOML file (must open in binary mode)
with open('output.toml', 'wb') as file:
    tomli_w.dump(config, file)

print("TOML configuration written")

Format	Pros	Cons	Best For
INI	Simple, widely supported	Limited nesting, manual type conversion	Simple configs
YAML	Clean syntax, good nesting	Indentation-sensitive, security concerns	Complex configs
TOML	Clear syntax, strong typing	Less known, limited Python support pre-3.11	Modern Python apps

Real-World Configuration Example

Configuration with Hierarchy and Validation

# config.py - Configuration management with hierarchy
import os
from dotenv import load_dotenv
from dataclasses import dataclass
from typing import Optional

# Load .env file
load_dotenv()

@dataclass
class DatabaseConfig:
    """Database configuration"""
    host: str = 'localhost'
    port: int = 5432
    name: str = 'myapp'
    user: str = 'postgres'
    password: str = ''

    @property
    def url(self) -> str:
        """Build database URL"""
        return f"postgresql://{self.user}:{self.password}@{self.host}:{self.port}/{self.name}"

    @classmethod
    def from_env(cls):
        """Load from environment variables"""
        return cls(
            host=os.getenv('DB_HOST', cls.host),
            port=int(os.getenv('DB_PORT', cls.port)),
            name=os.getenv('DB_NAME', cls.name),
            user=os.getenv('DB_USER', cls.user),
            password=os.getenv('DB_PASSWORD', cls.password)
        )

@dataclass
class AppConfig:
    """Application configuration"""
    debug: bool = False
    log_level: str = 'INFO'
    secret_key: str = 'change-me-in-production'
    allowed_hosts: list = None

    def __post_init__(self):
        if self.allowed_hosts is None:
            self.allowed_hosts = ['localhost']

    @classmethod
    def from_env(cls):
        """Load from environment variables"""
        debug = os.getenv('DEBUG', 'False').lower() in ('true', '1', 'yes')
        allowed_hosts = os.getenv('ALLOWED_HOSTS', 'localhost').split(',')

        return cls(
            debug=debug,
            log_level=os.getenv('LOG_LEVEL', cls.log_level),
            secret_key=os.getenv('SECRET_KEY', cls.secret_key),
            allowed_hosts=allowed_hosts
        )

    def validate(self):
        """Validate configuration"""
        if self.secret_key == 'change-me-in-production':
            raise ValueError("SECRET_KEY must be changed in production")
        if self.debug and 'production' in os.getenv('ENVIRONMENT', ''):
            raise ValueError("DEBUG must be False in production")

@dataclass
class Config:
    """Main configuration class"""
    database: DatabaseConfig
    app: AppConfig

    @classmethod
    def load(cls):
        """Load complete configuration"""
        config = cls(
            database=DatabaseConfig.from_env(),
            app=AppConfig.from_env()
        )

        # Validate in production
        if not config.app.debug:
            config.app.validate()

        return config


# Usage
if __name__ == '__main__':
    config = Config.load()

    print(f"Database URL: {config.database.url}")
    print(f"Debug mode: {config.app.debug}")
    print(f"Log level: {config.app.log_level}")
    print(f"Allowed hosts: {config.app.allowed_hosts}")

Configuration Best Practices

DO

Use environment variables for secrets
Provide sensible defaults
Validate configuration on startup
Document all configuration options
Use different configs per environment
Keep config separate from code

DON'T

Hardcode secrets in code
Commit .env or secrets to git
Use production configs in development
Ignore missing required config
Mix configuration formats unnecessarily
Leave default passwords in production

Security Checklist

✓ Never commit secrets: Add .env, config.ini (with secrets) to .gitignore
✓ Use .env.example: Template file without actual secrets
✓ Rotate secrets regularly: Change API keys, passwords periodically
✓ Different secrets per environment: Dev ≠ staging ≠ production
✓ Use secret managers: AWS Secrets Manager, Azure Key Vault, etc.
✓ Validate on load: Fail fast if configuration is invalid

Key Takeaways

Separate configuration from code for flexibility and security
Environment variables are perfect for secrets and environment-specific settings
.env files with python-dotenv simplify local development
configparser handles INI files natively in Python
YAML is excellent for complex, nested configurations
TOML offers strong typing and is great for modern Python projects
Use configuration hierarchy: defaults → files → env vars → CLI args
Never commit secrets to version control - use .gitignore
Validate configuration on application startup to fail fast
Document all options in .env.example and README

What's Next?

You've mastered configuration management! Now let's explore SQLite and learn how to persist data with a lightweight, embedded database.

SQLite Basics - Work with a file-based relational database built into Python
SQL Operations - Create tables, query data, and manage transactions
Best Practices - Learn connection management, security, and performance optimization

Back to "CLI Tools"

Python IntermediateLesson 16

Next: "Database Basics (SQLite)"