Packaging & Distribution
Transform your Python code into professional, shareable packages
Why Packaging Matters
Packaging transforms Python code into something that can be installed, reused, versioned, and shared reliably. Without proper packaging, projects become fragile, hard to deploy, and difficult to maintain. Professional Python developers treat packaging as a core part of software design, not an afterthought.
Benefits of Proper Packaging:
- Easy installation: One command to install anywhere
- Dependency management: Automatic resolution of requirements
- Version control: Track changes and enable upgrades
- Distribution: Share via PyPI, Git, or private repositories
- Professionalism: Shows software engineering maturity
Python Packaging Evolution
Legacy
setup.pyTransitional
setup.cfgModern
pyproject.tomlRecommended ✓
pyproject.tomlpyproject.toml for all new projects. It's the official, declarative standard for Python packaging.Package Structure & Layout
A well-structured package follows conventions that make it easy to understand, test, and maintain.
A complete, production-ready package structure with all common components.
my-awesome-package/
├── .gitignore # Git ignore patterns
├── .github/
│ └── workflows/
│ └── ci.yml # GitHub Actions CI/CD
├── LICENSE # Software license (MIT, Apache, etc.)
├── README.md # Project overview and docs
├── pyproject.toml # Package metadata and config
├── src/ # Source code directory
│ └── my_package/
│ ├── __init__.py # Package initialization
│ ├── __main__.py # Entry point for python -m
│ ├── core.py # Core functionality
│ ├── utils.py # Utility functions
│ └── cli.py # Command-line interface
├── tests/ # Test directory
│ ├── __init__.py
│ ├── test_core.py
│ ├── test_utils.py
│ └── conftest.py # pytest configuration
├── docs/ # Documentation
│ ├── conf.py
│ └── index.md
└── examples/ # Usage examples
└── basic_usage.pypyproject.toml: Complete Guide
The pyproject.toml file is the central configuration file for your package. Let's explore all the important sections.
Complete pyproject.toml Example
# Build system configuration (required)
[build-system]
requires = ["setuptools>=68", "wheel"]
build-backend = "setuptools.build_meta"
# Project metadata (required)
[project]
name = "my-awesome-package"
version = "0.1.0"
description = "A fantastic Python package that does amazing things"
readme = "README.md"
requires-python = ">=3.10"
license = {text = "MIT"}
authors = [
{name = "Your Name", email = "you@example.com"},
]
maintainers = [
{name = "Maintainer Name", email = "maintainer@example.com"},
]
keywords = ["api", "data", "automation"]
classifiers = [
"Development Status :: 4 - Beta",
"Intended Audience :: Developers",
"License :: OSI Approved :: MIT License",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
]
# Dependencies
dependencies = [
"requests>=2.31.0,<3.0",
"pydantic>=2.0",
"click>=8.0",
]
# Optional dependencies (extras)
[project.optional-dependencies]
dev = [
"pytest>=7.0",
"pytest-cov>=4.0",
"black>=23.0",
"ruff>=0.1.0",
"mypy>=1.0",
]
docs = [
"mkdocs>=1.5",
"mkdocs-material>=9.0",
]
all = ["my-awesome-package[dev,docs]"]
# URLs
[project.urls]
Homepage = "https://github.com/username/my-awesome-package"
Documentation = "https://my-awesome-package.readthedocs.io"
Repository = "https://github.com/username/my-awesome-package"
Changelog = "https://github.com/username/my-awesome-package/blob/main/CHANGELOG.md"
"Bug Tracker" = "https://github.com/username/my-awesome-package/issues"
# CLI scripts
[project.scripts]
myapp = "my_package.cli:main"
myapp-admin = "my_package.cli:admin_main"
# Entry points (for plugins)
[project.entry-points."my_package.plugins"]
plugin_a = "my_package.plugins:PluginA"
# Tool configurations
[tool.setuptools]
packages = ["my_package"]
[tool.setuptools.package-dir]
"" = "src"
[tool.pytest.ini_options]
testpaths = ["tests"]
python_files = ["test_*.py"]
python_classes = ["Test*"]
python_functions = ["test_*"]
addopts = "-v --cov=my_package --cov-report=html"
[tool.black]
line-length = 100
target-version = ["py310", "py311", "py312"]
[tool.ruff]
line-length = 100
select = ["E", "F", "I"]
ignore = ["E501"]
[tool.mypy]
python_version = "3.10"
warn_return_any = true
warn_unused_configs = true
disallow_untyped_defs = trueSemantic Versioning (SemVer)
Semantic Versioning is the standard for communicating changes in your package. Format: MAJOR.MINOR.PATCH
2
MAJOR
Breaking changes
Incompatible API updates
5
MINOR
New features
Backwards compatible
3
PATCH
Bug fixes only
No new features
# Version progression examples 1.0.0 → 1.0.1 # Bug fix: Fixed login error 1.0.1 → 1.1.0 # New feature: Added export to PDF 1.1.0 → 1.1.1 # Bug fix: Fixed PDF formatting 1.1.1 → 2.0.0 # Breaking: Changed API authentication method # Pre-release versions 1.0.0-alpha.1 # Alpha release 1.0.0-beta.2 # Beta release 1.0.0-rc.1 # Release candidate # Development versions 1.0.0.dev1 # Development version
Dependency Management Best Practices
Managing dependencies correctly is crucial for package stability and compatibility.
Dependency Specification Formats
dependencies = [
# Minimum version (flexible)
"requests>=2.31.0",
# Version range (recommended for libraries)
"pydantic>=2.0,<3.0",
# Compatible release (recommended)
"flask~=3.0.0", # Allows 3.0.x but not 3.1.0
# Exact version (avoid unless necessary)
"numpy==1.24.3",
# Multiple constraints
"django>=4.2,<5.0,!=4.2.5", # Exclude specific version
# Optional dependencies with extras
"sqlalchemy[asyncio]>=2.0",
# Environment markers
'pywin32>=300; platform_system == "Windows"',
'uvloop>=0.17; sys_platform != "win32"',
]✓ Good Practices
- Specify minimum versions
- Use version ranges for libraries
- Keep dependencies minimal
- Use optional dependencies for extras
- Document why each dependency exists
✗ Bad Practices
- No version constraints (risky)
- Exact pins for libraries (too rigid)
- Including dev tools as dependencies
- Depending on alpha/beta versions
- Circular dependencies
Optional Dependencies (Extras)
[project.optional-dependencies]
# Development tools
dev = [
"pytest>=7.0",
"black>=23.0",
"mypy>=1.0",
]
# Documentation
docs = [
"mkdocs>=1.5",
"mkdocs-material>=9.0",
]
# Database support
postgres = ["psycopg2-binary>=2.9"]
mysql = ["pymysql>=1.0"]
# All extras
all = ["mypackage[dev,docs,postgres,mysql]"]
# Install with extras:
# pip install mypackage[dev]
# pip install mypackage[postgres,docs]
# pip install mypackage[all]Building and Installing Packages
Local Development Installation
# Editable install (development mode) pip install -e . # With extras pip install -e ".[dev]" # Why editable install? # - Changes to source code are immediately available # - No need to reinstall after each change # - Perfect for active development
Building Distribution Packages
# Install build tool pip install build # Build source distribution and wheel python -m build # Output: # dist/ # ├── my_package-0.1.0.tar.gz # Source distribution # └── my_package-0.1.0-py3-none-any.whl # Wheel (faster install) # Install from wheel pip install dist/my_package-0.1.0-py3-none-any.whl # Install from source distribution pip install dist/my_package-0.1.0.tar.gz
- Wheel (.whl): Pre-built, faster to install, platform-specific for compiled extensions
- Source (.tar.gz): Requires building during install, works everywhere
Command-Line Interface (CLI) Entry Points
Make your package executable from the command line by defining entry points.
# pyproject.toml
[project.scripts]
myapp = "my_package.cli:main"
myapp-admin = "my_package.cli:admin"
# src/my_package/cli.py
import click
@click.command()
@click.option('--name', default='World', help='Name to greet')
@click.option('--count', default=1, help='Number of greetings')
def main(name, count):
"""Simple CLI tool"""
for _ in range(count):
click.echo(f"Hello {name}!")
@click.command()
def admin():
"""Admin commands"""
click.echo("Admin mode activated")
if __name__ == '__main__':
main()
# After installation:
# $ myapp --name Alice --count 3
# Hello Alice!
# Hello Alice!
# Hello Alice!
#
# $ myapp-admin
# Admin mode activatedPublishing to PyPI
The Python Package Index (PyPI) is the official repository for Python packages. Publishing makes your package available via pip install.
Step-by-Step Publishing
# 1. Create accounts # - Test PyPI: https://test.pypi.org/account/register/ # - Real PyPI: https://pypi.org/account/register/ # 2. Install publishing tools pip install build twine # 3. Build your package python -m build # 4. Upload to Test PyPI first twine upload --repository testpypi dist/* # 5. Test installation from Test PyPI pip install --index-url https://test.pypi.org/simple/ my-package # 6. If everything works, upload to real PyPI twine upload dist/* # 7. Install from PyPI pip install my-package
Using API Tokens (Recommended)
# Create ~/.pypirc file
[distutils]
index-servers =
pypi
testpypi
[pypi]
username = __token__
password = pypi-AgEIcHlwaS5vcmc... # Your API token
[testpypi]
username = __token__
password = pypi-AgENdGVzdC5weXBp... # Your Test PyPI token
# Now you can upload without entering credentials
twine upload dist/*- Choose a unique package name (check pypi.org)
- Add a comprehensive README.md
- Include a LICENSE file
- Test thoroughly on Test PyPI first
- Version numbers cannot be reused - plan carefully!
Alternative Distribution Methods
Git Installation
Install directly from Git repositories
# From GitHub pip install git+https://github.com/user/repo.git # Specific branch pip install git+https://github.com/user/repo.git@main # Specific tag/version pip install git+https://github.com/user/repo.git@v1.0.0 # With extras pip install "mypackage[dev] @ git+https://github.com/user/repo.git"
Private Repositories
Host private packages internally
# Using private PyPI server pip install --index-url https://pypi.company.com/simple/ mypackage # Using artifacts repository (JFrog, Nexus) pip install --extra-index-url https://artifacts.company.com/pypi/ mypackage # From network drive pip install /path/to/mypackage-1.0.0.whl
CI/CD Integration
Automate building, testing, and publishing with continuous integration.
# .github/workflows/publish.yml
name: Publish to PyPI
on:
release:
types: [published]
jobs:
publish:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.11'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install build twine
- name: Build package
run: python -m build
- name: Publish to PyPI
env:
TWINE_USERNAME: __token__
TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
run: twine upload dist/*- Consistent builds across environments
- Automatic testing before publishing
- Version tagging and changelog generation
- Reduced human error in releases
Packaging Best Practices
✓ Do This
- Use
src/layout for projects - Write comprehensive README.md
- Include LICENSE file
- Follow semantic versioning
- Specify dependency ranges appropriately
- Test on multiple Python versions
- Use pyproject.toml for configuration
- Document breaking changes
- Automate releases with CI/CD
✗ Avoid This
- Using legacy setup.py for new projects
- Unpinned or overly strict dependencies
- Publishing without testing
- Breaking changes without major version bump
- Missing or incomplete documentation
- No LICENSE file
- Inconsistent naming conventions
- Including secrets in package
- Manual, error-prone release process
Modern Packaging Tools
While standard tools like pip and venv are sufficient for many projects, newer tools like Poetry and UV offer advanced features for dependency management and speed.
Poetry
Does everything: manages dependencies, builds packages, and publishes to PyPI.
- Pros: Excellent dependency resolver, easy publishing flow, intuitive CLI.
- Cons: Non-standard
poetry.lockformat, slower than UV.
UV (by Astral)
An extremely fast Python package installer and resolver, written in Rust.
- Pros: Blazing fast (10-100x pip), fully compatible with pip/venv.
- Cons: Newer tool ecosystem, simplified feature set compared to Poetry.
Real World Advice: What should I use?
Modern tools are great, but standard pyproject.toml + pip is often "just good enough".
Use Standard (pip + venv) when:
- Deploying single services (Lambda, ECS tasks)
- Working with Docker (keeps images simple)
- You want maximum compatibility/stability
Consider Poetry/UV when:
- Developing complex libraries with many devs
- You need strict dependency locking
- Speed is a critical bottleneck (use UV)
Evaluation: The "Standardist" Approach
Strengths
- Reduced Complexity: Avoids layers of abstraction. If a tool like Poetry breaks, your pipeline remains unaffected.
- PEP 621 Compliance: Using
pyproject.tomlis the future-proof official standard.
Potential Risks
- Lockfile Absence: Standard
pyproject.tomldoesn't inherently lock versions, risking "it works on my machine" bugs. - Resolution Speed:
pipis significantly slower than UV at resolving complex dependency trees.
While modern wrappers like Poetry and UV offer impressive resolution speeds and solve specific dependency conflicts, they often introduce an unnecessary layer of abstraction. For an experienced team that understands environment isolation and dependency pinning, the native Python ecosystem is robust enough, then you can prioritize lean, standards-compliant builds over third-party management layers that can complicate the CI/CD pipeline.
Key Takeaways
- Modern packaging uses pyproject.toml - it's the official standard
- src/ layout prevents import issues - use it for professional projects
- Semantic versioning communicates changes - never break compatibility without warning
- Manage dependencies carefully - balance flexibility and stability
- Test before publishing - use Test PyPI first
- Automate with CI/CD - reduce errors and save time
- Documentation matters - README and LICENSE are essential
- Distribution options are flexible - PyPI, Git, or private repos
Practice Exercises
Exercise 1: Create a Package
Build a simple utility package with src/ layout, pyproject.toml, and a CLI command. Install it in editable mode and test the CLI works. Include at least 3 functions and write tests for them.
Exercise 2: Publish to Test PyPI
Take your package from Exercise 1, add proper documentation (README, LICENSE), build distributions, and publish to Test PyPI. Then install it from Test PyPI in a clean virtual environment to verify it works.
Exercise 3: Add CI/CD
Set up GitHub Actions to automatically run tests on push, and automatically publish to Test PyPI when you create a release. Test the full workflow by creating a new version tag.
Additional Resources
- Official Docs: packaging.python.org - comprehensive packaging guide
- PEP 621: pyproject.toml standard specification
- Sample Projects: github.com/pypa/sampleproject
- Tools: cookiecutter-pypackage for project templates
- Testing: tox for testing across Python versions
- Advanced: Poetry, PDM for alternative package management
What's Next?
You can now package your code! Let's add type hints and linting to make your code more maintainable.
- Type Annotations - Add type hints for better code documentation
- Type Checkers - Use mypy to catch type errors before runtime
- Code Linters - Enforce code quality with pylint and flake8