Dependency Management Guide

Overview

This document explains how Python dependencies are managed in this project, ensuring consistency across development environments, CI/CD pipelines, and production deployments.

Single Source of Truth: pyproject.toml

The authoritative source for all Python dependencies is pyproject.toml. All other dependency files should be derived from or kept in sync with this file.

Structure

[project]
dependencies = [...]           # Production runtime dependencies

[project.optional-dependencies]
dev = [...]                    # Development and testing tools
fast-io = [...]               # Optional performance dependencies

Dependency Files

1. pyproject.toml

• Purpose: Primary dependency specification
• Contains:
• Core runtime dependencies ([project.dependencies])
• Optional dev dependencies ([project.optional-dependencies.dev])
• Optional fast-io dependencies ([project.optional-dependencies.fast-io])
• When to update: When adding, removing, or changing version constraints for dependencies

2. requirements.txt

• Purpose: Production runtime dependencies for pip installation
• Source: Should match [project.dependencies] in pyproject.toml
• Used by:
• GitHub Actions workflows
• Dev container setup (.devcontainer/postCreate.sh)
• Docker images (if applicable)
• Production deployments

3. requirements-dev.txt

• Purpose: Development and testing dependencies
• Source: Should match [project.optional-dependencies.dev] in pyproject.toml
• Used by:
• GitHub Actions workflows
• Dev container setup
• Local development environments
• Pre-commit hooks

4. .pre-commit-config.yaml

• Purpose: Pre-commit hook configuration with pinned versions
• Version strategy: Exact versions (e.g., v0.14.2) for reproducibility
• Relationship to requirements:
• Pre-commit runs in isolated environments
• Pinned versions ensure consistent behavior across all developers
• Requirements files specify minimum versions (e.g., >=0.8.0)
• This difference is intentional and correct

Version Pinning Philosophy

Requirements Files (requirements*.txt)

• Use minimum version constraints: >=X.Y.Z
• Allow upgrades within compatible ranges: >=2.3,<3.0
• Reason: Flexibility for security updates and compatible improvements

Pre-commit Hooks (.pre-commit-config.yaml)

• Use exact version pins: rev: v0.14.2
• Reason: Reproducibility across all environments and developers

Example

# pyproject.toml - minimum version
[project.optional-dependencies]
dev = ["ruff>=0.8.0"]  # Allow any version >= 0.8.0

# .pre-commit-config.yaml - exact version
- repo: https://github.com/astral-sh/ruff-pre-commit
  rev: v0.14.2  # Exact version for reproducibility

Licensing Policy

• Prefer permissive licenses (MIT/BSD/Apache-2.0) for runtime dependencies to ensure long-term viability and ease of redistribution.
• Avoid copyleft dependencies for runtime unless critically necessary and license obligations are acceptable.
• Periodically review licenses of transitive dependencies when upgrading; document exceptions with rationale.

Installation Patterns

Local Development (Dev Container)

pip install --user -r requirements.txt
pip install --user -r requirements-dev.txt
pip install --user -e .

GitHub Actions

- name: Install dependencies
  run: |
    python -m pip install --upgrade pip
    python -m pip install -r requirements.txt
    python -m pip install -r requirements-dev.txt
    python -m pip install -e .

Production (Runtime Only)

pip install -r requirements.txt
pip install -e .

Updating Dependencies

Adding a New Dependency

1. Add to pyproject.toml (primary source):

[project]
dependencies = [
    "new-package>=1.0.0",
]

1. Update requirements.txt to match:

new-package>=1.0.0

1. If it's a dev dependency, add to both:

2. [project.optional-dependencies.dev] in pyproject.toml

3. requirements-dev.txt

4. If used in pre-commit, add to .pre-commit-config.yaml with exact version:

- repo: https://github.com/owner/repo
  rev: v1.0.0  # Exact version

1. Test installation:

pip install -r requirements.txt
pip install -r requirements-dev.txt
pip install -e .

1. Verify in CI: Push and ensure all GitHub Actions pass

Upgrading a Dependency

1. Update version in pyproject.toml
2. Update matching requirements*.txt file
3. If in pre-commit, update to new pinned version
4. Test locally and in CI

Removing a Dependency

1. Remove from pyproject.toml
2. Remove from matching requirements*.txt file
3. Remove from .pre-commit-config.yaml if present
4. Test to ensure nothing breaks

Consistency Validation

Manual Check

# Check for duplicates in requirements.txt
sort requirements.txt | uniq -d

# Compare with pyproject.toml
diff <(grep -E '^\s+"[^"]+"' pyproject.toml | sort) \
     <(sed 's/>=.*$//' requirements.txt | sort)

Automated Validation (Future Enhancement)

Consider adding a script to CI that validates:

• No duplicate entries in requirements files
• All pyproject.toml dependencies are in requirements files
• No missing dependencies

Common Pitfalls

❌ Don't: Install packages globally without updating configs

pip install some-package  # Wrong! Not tracked

✅ Do: Add to pyproject.toml first, then install

# 1. Edit pyproject.toml and requirements.txt
# 2. Then install
pip install -r requirements.txt

❌ Don't: Mix different version constraints

# requirements.txt
pandas>=2.3,<3.0

# pyproject.toml
pandas>=2.0,<3.0  # Inconsistent!

✅ Do: Keep version constraints synchronized

Both files should specify the same version range.

❌ Don't: Use exact pins in requirements files

pandas==2.3.0  # Too restrictive for requirements.txt

✅ Do: Use minimum version with upper bounds for major versions

pandas>=2.3,<3.0  # Allows patch/minor updates

GitHub Actions Consistency

All workflows use the same installation pattern:

1. Upgrade pip
2. Install requirements.txt (production dependencies)
3. Install requirements-dev.txt (dev/test dependencies)
4. Install package in editable mode (-e .)

Exception: Architecture and Type Check

These workflows only install requirements-dev.txt and -e . for speed. The package dependencies are available through the editable install.

Dev Container Setup

The .devcontainer/postCreate.sh script:

1. Installs requirements.txt
2. Installs requirements-dev.txt
3. Installs package in editable mode
4. Installs pre-commit hooks
5. Creates test fixture directories

This ensures the dev container matches CI/CD environments exactly.

Python Version

All environments use Python 3.12:

• Dev container: python3.12
• GitHub Actions: python-version: "3.12"
• pyproject.toml: requires-python = ">=3.10" (minimum)

The minimum is set to 3.10 for broader compatibility, but development and CI use 3.12.

Tool-Specific Notes

Ruff (Formatter and Linter)

• Requirements: ruff>=0.8.0 (minimum)
• Pre-commit: rev: v0.14.2 (exact)
• Configuration: pyproject.toml [tool.ruff] section
• Replaces: black, isort, flake8, pylint, and many others

Mypy (Type Checker)

• Requirements: mypy>=1.11.0
• Configuration: mypy.ini
• Type stubs required:
• pandas-stubs>=2.2.0
• types-PyYAML>=6.0.12
• types-tqdm

Pytest (Testing)

• Requirements: pytest>=8.0.0
• Plugins:
• pytest-xdist>=3.5.0 (parallel execution)
• pytest-timeout>=2.2.0 (timeout handling)
• pytest-randomly>=3.15.0 (randomized test order)
• pytest-benchmark>=4.0.0 (performance testing)

Pre-commit

• Requirements: pre-commit>=4.0.0
• Configuration: .pre-commit-config.yaml
• Automatically installed in dev container
• Runs on every commit to catch issues early

Troubleshooting

"No module named X" Error

1. Check if X is in requirements.txt or requirements-dev.txt
2. If missing, add to appropriate file and pyproject.toml
3. Reinstall: pip install -r requirements-dev.txt

Pre-commit Hook Failing

1. Check .pre-commit-config.yaml for correct versions
2. Update hooks: pre-commit autoupdate
3. Clean cache: pre-commit clean
4. Reinstall: pre-commit install

GitHub Actions Failing, Local Tests Pass

1. Verify requirements.txt and requirements-dev.txt are up to date
2. Check for system-specific dependencies
3. Review workflow YAML for correct installation steps
4. Ensure Python 3.12 is used locally: python --version

Dependency Conflicts

1. Check for version incompatibilities in requirements.txt
2. Use pip install --dry-run to simulate
3. Consider relaxing version constraints if too restrictive
4. Update conflicting packages together

Best Practices

1. Always update pyproject.toml first - It's the source of truth
2. Keep requirements files in sync with pyproject.toml
3. Test locally before pushing to catch issues early
4. Use virtual environments or the dev container for isolation
5. Pin versions in pre-commit for reproducibility
6. Use minimum versions in requirements for flexibility
7. Document why when using unusual version constraints
8. Run pre-commit before committing: pre-commit run --all-files
9. Keep dev and CI in sync - same Python version, same packages
10. Review dependency updates regularly for security patches

Future Enhancements

Consider pip-compile from pip-tools

# Generate requirements.txt from pyproject.toml
pip-compile pyproject.toml -o requirements.txt

# Generate requirements-dev.txt with dev extras
pip-compile --extra dev pyproject.toml -o requirements-dev.txt

Benefits:

• Automatic synchronization
• Generates lock files with exact versions
• Resolves dependency conflicts automatically
• Makes updating dependencies easier

Automated Validation Script

Create a script to validate configuration consistency:

# scripts/validate_dependencies.py
# Check for:
# - Duplicate entries
# - Mismatches between files
# - Missing dependencies
# - Version constraint inconsistencies

Add to pre-commit or GitHub Actions for automatic validation.

References

• PEP 621: Project Metadata
• pip-tools Documentation
• Pre-commit Framework
• Ruff Documentation
• GitHub Actions Python Setup