Packaging & Distribution

Transform your Python code into professional, shareable packages

Why Packaging Matters

Packaging transforms Python code into something that can be installed, reused, versioned, and shared reliably. Without proper packaging, projects become fragile, hard to deploy, and difficult to maintain. Professional Python developers treat packaging as a core part of software design, not an afterthought.

Benefits of Proper Packaging:

  • Easy installation: One command to install anywhere
  • Dependency management: Automatic resolution of requirements
  • Version control: Track changes and enable upgrades
  • Distribution: Share via PyPI, Git, or private repositories
  • Professionalism: Shows software engineering maturity

Python Packaging Evolution

Legacy
setup.py
Transitional
setup.cfg
Modern
pyproject.toml
Recommended ✓
pyproject.toml
Modern standard (PEP 621): Use pyproject.toml for all new projects. It's the official, declarative standard for Python packaging.

Package Structure & Layout

A well-structured package follows conventions that make it easy to understand, test, and maintain.

A complete, production-ready package structure with all common components.

my-awesome-package/
├── .gitignore                 # Git ignore patterns
├── .github/
│   └── workflows/
│       └── ci.yml             # GitHub Actions CI/CD
├── LICENSE                    # Software license (MIT, Apache, etc.)
├── README.md                  # Project overview and docs
├── pyproject.toml             # Package metadata and config
├── src/                       # Source code directory
│   └── my_package/
│       ├── __init__.py        # Package initialization
│       ├── __main__.py        # Entry point for python -m
│       ├── core.py            # Core functionality
│       ├── utils.py           # Utility functions
│       └── cli.py             # Command-line interface
├── tests/                     # Test directory
│   ├── __init__.py
│   ├── test_core.py
│   ├── test_utils.py
│   └── conftest.py            # pytest configuration
├── docs/                      # Documentation
│   ├── conf.py
│   └── index.md
└── examples/                  # Usage examples
    └── basic_usage.py

pyproject.toml: Complete Guide

The pyproject.toml file is the central configuration file for your package. Let's explore all the important sections.

Complete pyproject.toml Example
# Build system configuration (required)
[build-system]
requires = ["setuptools>=68", "wheel"]
build-backend = "setuptools.build_meta"

# Project metadata (required)
[project]
name = "my-awesome-package"
version = "0.1.0"
description = "A fantastic Python package that does amazing things"
readme = "README.md"
requires-python = ">=3.10"
license = {text = "MIT"}
authors = [
    {name = "Your Name", email = "you@example.com"},
]
maintainers = [
    {name = "Maintainer Name", email = "maintainer@example.com"},
]
keywords = ["api", "data", "automation"]
classifiers = [
    "Development Status :: 4 - Beta",
    "Intended Audience :: Developers",
    "License :: OSI Approved :: MIT License",
    "Programming Language :: Python :: 3",
    "Programming Language :: Python :: 3.10",
    "Programming Language :: Python :: 3.11",
    "Programming Language :: Python :: 3.12",
]

# Dependencies
dependencies = [
    "requests>=2.31.0,<3.0",
    "pydantic>=2.0",
    "click>=8.0",
]

# Optional dependencies (extras)
[project.optional-dependencies]
dev = [
    "pytest>=7.0",
    "pytest-cov>=4.0",
    "black>=23.0",
    "ruff>=0.1.0",
    "mypy>=1.0",
]
docs = [
    "mkdocs>=1.5",
    "mkdocs-material>=9.0",
]
all = ["my-awesome-package[dev,docs]"]

# URLs
[project.urls]
Homepage = "https://github.com/username/my-awesome-package"
Documentation = "https://my-awesome-package.readthedocs.io"
Repository = "https://github.com/username/my-awesome-package"
Changelog = "https://github.com/username/my-awesome-package/blob/main/CHANGELOG.md"
"Bug Tracker" = "https://github.com/username/my-awesome-package/issues"

# CLI scripts
[project.scripts]
myapp = "my_package.cli:main"
myapp-admin = "my_package.cli:admin_main"

# Entry points (for plugins)
[project.entry-points."my_package.plugins"]
plugin_a = "my_package.plugins:PluginA"

# Tool configurations
[tool.setuptools]
packages = ["my_package"]

[tool.setuptools.package-dir]
"" = "src"

[tool.pytest.ini_options]
testpaths = ["tests"]
python_files = ["test_*.py"]
python_classes = ["Test*"]
python_functions = ["test_*"]
addopts = "-v --cov=my_package --cov-report=html"

[tool.black]
line-length = 100
target-version = ["py310", "py311", "py312"]

[tool.ruff]
line-length = 100
select = ["E", "F", "I"]
ignore = ["E501"]

[tool.mypy]
python_version = "3.10"
warn_return_any = true
warn_unused_configs = true
disallow_untyped_defs = true

Semantic Versioning (SemVer)

Semantic Versioning is the standard for communicating changes in your package. Format: MAJOR.MINOR.PATCH

2

MAJOR

Breaking changes
Incompatible API updates

5

MINOR

New features
Backwards compatible

3

PATCH

Bug fixes only
No new features

# Version progression examples

1.0.0  →  1.0.1  # Bug fix: Fixed login error
1.0.1  →  1.1.0  # New feature: Added export to PDF
1.1.0  →  1.1.1  # Bug fix: Fixed PDF formatting
1.1.1  →  2.0.0  # Breaking: Changed API authentication method

# Pre-release versions
1.0.0-alpha.1    # Alpha release
1.0.0-beta.2     # Beta release
1.0.0-rc.1       # Release candidate

# Development versions
1.0.0.dev1       # Development version
The Golden Rule: Never break user code without bumping the MAJOR version. If you must make breaking changes, document them clearly in CHANGELOG.md.

Dependency Management Best Practices

Managing dependencies correctly is crucial for package stability and compatibility.

Dependency Specification Formats
dependencies = [
    # Minimum version (flexible)
    "requests>=2.31.0",

    # Version range (recommended for libraries)
    "pydantic>=2.0,<3.0",

    # Compatible release (recommended)
    "flask~=3.0.0",  # Allows 3.0.x but not 3.1.0

    # Exact version (avoid unless necessary)
    "numpy==1.24.3",

    # Multiple constraints
    "django>=4.2,<5.0,!=4.2.5",  # Exclude specific version

    # Optional dependencies with extras
    "sqlalchemy[asyncio]>=2.0",

    # Environment markers
    'pywin32>=300; platform_system == "Windows"',
    'uvloop>=0.17; sys_platform != "win32"',
]
✓ Good Practices
  • Specify minimum versions
  • Use version ranges for libraries
  • Keep dependencies minimal
  • Use optional dependencies for extras
  • Document why each dependency exists
✗ Bad Practices
  • No version constraints (risky)
  • Exact pins for libraries (too rigid)
  • Including dev tools as dependencies
  • Depending on alpha/beta versions
  • Circular dependencies
Optional Dependencies (Extras)
[project.optional-dependencies]
# Development tools
dev = [
    "pytest>=7.0",
    "black>=23.0",
    "mypy>=1.0",
]

# Documentation
docs = [
    "mkdocs>=1.5",
    "mkdocs-material>=9.0",
]

# Database support
postgres = ["psycopg2-binary>=2.9"]
mysql = ["pymysql>=1.0"]

# All extras
all = ["mypackage[dev,docs,postgres,mysql]"]

# Install with extras:
# pip install mypackage[dev]
# pip install mypackage[postgres,docs]
# pip install mypackage[all]

Building and Installing Packages

Local Development Installation
# Editable install (development mode)
pip install -e .

# With extras
pip install -e ".[dev]"

# Why editable install?
# - Changes to source code are immediately available
# - No need to reinstall after each change
# - Perfect for active development
Building Distribution Packages
# Install build tool
pip install build

# Build source distribution and wheel
python -m build

# Output:
# dist/
# ├── my_package-0.1.0.tar.gz        # Source distribution
# └── my_package-0.1.0-py3-none-any.whl  # Wheel (faster install)

# Install from wheel
pip install dist/my_package-0.1.0-py3-none-any.whl

# Install from source distribution
pip install dist/my_package-0.1.0.tar.gz
Wheel vs Source Distribution:
  • Wheel (.whl): Pre-built, faster to install, platform-specific for compiled extensions
  • Source (.tar.gz): Requires building during install, works everywhere
Always provide both for maximum compatibility.

Command-Line Interface (CLI) Entry Points

Make your package executable from the command line by defining entry points.

# pyproject.toml
[project.scripts]
myapp = "my_package.cli:main"
myapp-admin = "my_package.cli:admin"

# src/my_package/cli.py
import click

@click.command()
@click.option('--name', default='World', help='Name to greet')
@click.option('--count', default=1, help='Number of greetings')
def main(name, count):
    """Simple CLI tool"""
    for _ in range(count):
        click.echo(f"Hello {name}!")

@click.command()
def admin():
    """Admin commands"""
    click.echo("Admin mode activated")

if __name__ == '__main__':
    main()

# After installation:
# $ myapp --name Alice --count 3
# Hello Alice!
# Hello Alice!
# Hello Alice!
#
# $ myapp-admin
# Admin mode activated

Publishing to PyPI

The Python Package Index (PyPI) is the official repository for Python packages. Publishing makes your package available via pip install.

Step-by-Step Publishing
# 1. Create accounts
# - Test PyPI: https://test.pypi.org/account/register/
# - Real PyPI: https://pypi.org/account/register/

# 2. Install publishing tools
pip install build twine

# 3. Build your package
python -m build

# 4. Upload to Test PyPI first
twine upload --repository testpypi dist/*

# 5. Test installation from Test PyPI
pip install --index-url https://test.pypi.org/simple/ my-package

# 6. If everything works, upload to real PyPI
twine upload dist/*

# 7. Install from PyPI
pip install my-package
Using API Tokens (Recommended)
# Create ~/.pypirc file
[distutils]
index-servers =
    pypi
    testpypi

[pypi]
username = __token__
password = pypi-AgEIcHlwaS5vcmc...  # Your API token

[testpypi]
username = __token__
password = pypi-AgENdGVzdC5weXBp...  # Your Test PyPI token

# Now you can upload without entering credentials
twine upload dist/*
Before publishing:
  • Choose a unique package name (check pypi.org)
  • Add a comprehensive README.md
  • Include a LICENSE file
  • Test thoroughly on Test PyPI first
  • Version numbers cannot be reused - plan carefully!

Alternative Distribution Methods

Git Installation

Install directly from Git repositories

# From GitHub
pip install git+https://github.com/user/repo.git

# Specific branch
pip install git+https://github.com/user/repo.git@main

# Specific tag/version
pip install git+https://github.com/user/repo.git@v1.0.0

# With extras
pip install "mypackage[dev] @ git+https://github.com/user/repo.git"
Private Repositories

Host private packages internally

# Using private PyPI server
pip install --index-url https://pypi.company.com/simple/ mypackage

# Using artifacts repository (JFrog, Nexus)
pip install --extra-index-url https://artifacts.company.com/pypi/ mypackage

# From network drive
pip install /path/to/mypackage-1.0.0.whl

CI/CD Integration

Automate building, testing, and publishing with continuous integration.

# .github/workflows/publish.yml
name: Publish to PyPI

on:
  release:
    types: [published]

jobs:
  publish:
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v4

    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.11'

    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install build twine

    - name: Build package
      run: python -m build

    - name: Publish to PyPI
      env:
        TWINE_USERNAME: __token__
        TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
      run: twine upload dist/*
Automation benefits:
  • Consistent builds across environments
  • Automatic testing before publishing
  • Version tagging and changelog generation
  • Reduced human error in releases

Packaging Best Practices

✓ Do This
  • Use src/ layout for projects
  • Write comprehensive README.md
  • Include LICENSE file
  • Follow semantic versioning
  • Specify dependency ranges appropriately
  • Test on multiple Python versions
  • Use pyproject.toml for configuration
  • Document breaking changes
  • Automate releases with CI/CD
✗ Avoid This
  • Using legacy setup.py for new projects
  • Unpinned or overly strict dependencies
  • Publishing without testing
  • Breaking changes without major version bump
  • Missing or incomplete documentation
  • No LICENSE file
  • Inconsistent naming conventions
  • Including secrets in package
  • Manual, error-prone release process

Modern Packaging Tools

While standard tools like pip and venv are sufficient for many projects, newer tools like Poetry and UV offer advanced features for dependency management and speed.

Poetry

Does everything: manages dependencies, builds packages, and publishes to PyPI.

  • Pros: Excellent dependency resolver, easy publishing flow, intuitive CLI.
  • Cons: Non-standard poetry.lock format, slower than UV.
UV (by Astral)

An extremely fast Python package installer and resolver, written in Rust.

  • Pros: Blazing fast (10-100x pip), fully compatible with pip/venv.
  • Cons: Newer tool ecosystem, simplified feature set compared to Poetry.

Real World Advice: What should I use?

Modern tools are great, but standard pyproject.toml + pip is often "just good enough".

Use Standard (pip + venv) when:
  • Deploying single services (Lambda, ECS tasks)
  • Working with Docker (keeps images simple)
  • You want maximum compatibility/stability
Consider Poetry/UV when:
  • Developing complex libraries with many devs
  • You need strict dependency locking
  • Speed is a critical bottleneck (use UV)

Evaluation: The "Standardist" Approach

Strengths
  • Reduced Complexity: Avoids layers of abstraction. If a tool like Poetry breaks, your pipeline remains unaffected.
  • PEP 621 Compliance: Using pyproject.toml is the future-proof official standard.
Potential Risks
  • Lockfile Absence: Standard pyproject.toml doesn't inherently lock versions, risking "it works on my machine" bugs.
  • Resolution Speed: pip is significantly slower than UV at resolving complex dependency trees.

While modern wrappers like Poetry and UV offer impressive resolution speeds and solve specific dependency conflicts, they often introduce an unnecessary layer of abstraction. For an experienced team that understands environment isolation and dependency pinning, the native Python ecosystem is robust enough, then you can prioritize lean, standards-compliant builds over third-party management layers that can complicate the CI/CD pipeline.

Key Takeaways

  • Modern packaging uses pyproject.toml - it's the official standard
  • src/ layout prevents import issues - use it for professional projects
  • Semantic versioning communicates changes - never break compatibility without warning
  • Manage dependencies carefully - balance flexibility and stability
  • Test before publishing - use Test PyPI first
  • Automate with CI/CD - reduce errors and save time
  • Documentation matters - README and LICENSE are essential
  • Distribution options are flexible - PyPI, Git, or private repos

Practice Exercises

Exercise 1: Create a Package

Build a simple utility package with src/ layout, pyproject.toml, and a CLI command. Install it in editable mode and test the CLI works. Include at least 3 functions and write tests for them.

Exercise 2: Publish to Test PyPI

Take your package from Exercise 1, add proper documentation (README, LICENSE), build distributions, and publish to Test PyPI. Then install it from Test PyPI in a clean virtual environment to verify it works.

Exercise 3: Add CI/CD

Set up GitHub Actions to automatically run tests on push, and automatically publish to Test PyPI when you create a release. Test the full workflow by creating a new version tag.

Additional Resources

  • Official Docs: packaging.python.org - comprehensive packaging guide
  • PEP 621: pyproject.toml standard specification
  • Sample Projects: github.com/pypa/sampleproject
  • Tools: cookiecutter-pypackage for project templates
  • Testing: tox for testing across Python versions
  • Advanced: Poetry, PDM for alternative package management
What's Next?

You can now package your code! Let's add type hints and linting to make your code more maintainable.

  • Type Annotations - Add type hints for better code documentation
  • Type Checkers - Use mypy to catch type errors before runtime
  • Code Linters - Enforce code quality with pylint and flake8