Publishing Python Packages to PyPI with pyproject.toml

Have you ever written a useful Python utility and wished others could easily install it with pip install your-package? Publishing to PyPI (Python Package Index) makes that possible. In the past, packaging was a maze of setup.py, setup.cfg, and MANIFEST.in. But since PEP 517 and PEP 518, the community has standardized on pyproject.toml as the single source of truth for Python projects. This tutorial walks you through the entire process: from setting up your project structure, writing a robust pyproject.toml, building distributions, and uploading to PyPI. You'll also learn how to avoid common pitfalls, automate releases, and debug issues in production. By the end, you'll be able to share your code with the world confidently.

1. Project Structure and Prerequisites

Before publishing, your project must have a proper structure. A typical Python package looks like this:

`` my_package/ ├── pyproject.toml ├── README.md ├── LICENSE ├── src/ │ └── my_package/ │ ├── __init__.py │ └── core.py └── tests/ └── test_core.py ``

We recommend using a src/ layout to avoid import confusion. The pyproject.toml file is the heart of modern Python packaging. It replaces setup.py, setup.cfg, and MANIFEST.in. You'll need Python 3.7+ and the latest versions of build and twine installed:

``bash pip install build twine ``

Also, create a PyPI account at https://pypi.org and generate an API token for secure uploads.

[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"

[project]
name = "my-package"
version = "0.1.0"
authors = [
    {name="Your Name", email="you@example.com"}
]
description = "A short description of your package"
readme = "README.md"
requires-python = ">=3.7"
license = {text = "MIT"}
classifiers = [
    "Programming Language :: Python :: 3",
    "License :: OSI Approved :: MIT License",
    "Operating System :: OS Independent",
]

[project.urls]
Homepage = "https://github.com/yourname/my-package"

[tool.setuptools.packages.find]
where = ["src"]

2. Writing a Robust pyproject.toml

The pyproject.toml file uses TOML syntax. The [build-system] table declares the build backend (usually setuptools). The [project] table contains metadata: name, version, authors, description, readme, Python version requirements, license, and classifiers. Classifiers help users discover your package on PyPI. The [project.urls] table links to your project's homepage, repository, or documentation.

For package discovery, use [tool.setuptools.packages.find] with where = ["src"] if you use the src layout. If your package has dependencies, add a dependencies key:

``toml dependencies = [ "requests>=2.25.0", ] ``

You can also specify optional dependencies under [project.optional-dependencies].

[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"

[project]
name = "my-package"
version = "0.1.0"
description = "A useful utility for data processing"
readme = "README.md"
requires-python = ">=3.8"
license = {text = "MIT"}
authors = [
    {name = "Jane Doe", email = "jane@example.com"}
]
classifiers = [
    "Development Status :: 4 - Beta",
    "Intended Audience :: Developers",
    "Topic :: Software Development :: Libraries",
    "Programming Language :: Python :: 3.8",
    "Programming Language :: Python :: 3.9",
    "Programming Language :: Python :: 3.10",
    "License :: OSI Approved :: MIT License",
]
dependencies = [
    "numpy>=1.21.0",
    "pandas>=1.3.0",
]

[project.urls]
Homepage = "https://github.com/janedoe/my-package"
Documentation = "https://my-package.readthedocs.io"
Repository = "https://github.com/janedoe/my-package"

[tool.setuptools.packages.find]
where = ["src"]

3. Building Distributions

Once your pyproject.toml is ready, you can build your package into distributable artifacts. The modern way is to use the build package:

``bash python -m build ``

This creates two files in the dist/ directory: a source distribution (.tar.gz) and a wheel (.whl). The wheel is a pre-built distribution that installs faster. You can also build only the wheel with python -m build --wheel.

Before uploading, validate your distributions with twine check:

``bash twine check dist/* ``

This checks for common issues like missing metadata or broken README rendering. Fix any warnings before proceeding.

# Build distributions
python -m build

# Check distributions
twine check dist/*

4. Uploading to PyPI

To upload, you need a PyPI API token. Create one at https://pypi.org/manage/account/token/. Then use twine:

``bash twine upload dist/* ``

You'll be prompted for username and password. Use __token__ as username and your token as password. To avoid interactive prompts, you can set environment variables:

``bash export TWINE_USERNAME=__token__ export TWINE_PASSWORD=pypi-xxxxxxxx ``

For testing, use TestPyPI first:

``bash twine upload --repository testpypi dist/* ``

Then install from TestPyPI:

``bash pip install --index-url https://test.pypi.org/simple/ my-package ``

Once satisfied, upload to the real PyPI.

# Upload to TestPyPI
twine upload --repository testpypi dist/*

# Upload to PyPI
twine upload dist/*

5. Automating Releases with CI/CD

Manual publishing is error-prone. Automate with GitHub Actions or other CI tools. Here's a simple workflow that publishes to PyPI when you create a GitHub release:

```yaml name: Publish to PyPI

on: release: types: [published]

jobs: deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - name: Set up Python uses: actions/setup-python@v4 with: python-version: '3.10' - name: Install dependencies run: | python -m pip install --upgrade pip pip install build twine - name: Build run: python -m build - name: Publish env: TWINE_USERNAME: __token__ TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }} run: twine upload dist/* ```

Store your PyPI token in GitHub Secrets as PYPI_API_TOKEN. This workflow triggers on every release, ensuring consistent builds.

name: Publish to PyPI

on:
  release:
    types: [published]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.10'
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install build twine
    - name: Build
      run: python -m build
    - name: Publish
      env:
        TWINE_USERNAME: __token__
        TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
      run: twine upload dist/*

6. Best Practices and Common Pitfalls

Best Practices: - Use a src/ layout to prevent import confusion. - Include a comprehensive README with installation, usage, and examples. - Add a license file (MIT, Apache, etc.). - Use semantic versioning and document changes in a CHANGELOG. - Keep dependencies minimal and specify version ranges. - Test your package in a clean virtual environment before publishing.

Common Pitfalls: - Forgetting to include readme in pyproject.toml. - Using a package name that conflicts with an existing one. - Not cleaning the dist/ folder before rebuilding. - Uploading to PyPI without testing on TestPyPI. - Hardcoding API tokens in source code.

Debugging Tips: - Run twine check dist/* to catch metadata issues. - Use pip install -e . to test your package locally. - Check PyPI page after upload to ensure description renders correctly. - If upload fails with 400, check the error message; often it's a duplicate version or invalid name.

# Create a fresh virtual environment
python -m venv test_env
source test_env/bin/activate

# Install your package from TestPyPI
pip install --index-url https://test.pypi.org/simple/ my-package

# Test import
python -c "import my_package; print(my_package.__version__)"