Pre-commit Hooks for Python Projects: Automate Code Quality

Imagine you're working on a Python project with a team. Everyone writes code in their own style, occasionally forgets to run tests, or leaves behind unused imports. The result? Pull requests filled with formatting inconsistencies, linting errors, and broken tests. Fixing these issues manually is tedious and error-prone. Enter pre-commit hooks: a powerful tool that automates code quality checks before every commit. Pre-commit hooks are scripts that run automatically when you run git commit. They can check for syntax errors, enforce coding standards, run tests, and even fix formatting. By catching issues early, they save time, reduce code review friction, and maintain a clean codebase. In this tutorial, you'll learn how to set up pre-commit hooks for Python projects, configure popular hooks like black, flake8, and mypy, and integrate them into your workflow. We'll also cover debugging common issues and best practices for production use. By the end, you'll have a robust pre-commit setup that ensures every commit meets your team's quality standards.

What Are Pre-commit Hooks?

Pre-commit hooks are scripts that run automatically before each commit in a Git repository. They are part of the Git hook system, which allows you to trigger actions at various points in the Git lifecycle. For Python projects, pre-commit hooks are commonly used to enforce code quality standards, run linters, format code, check for security issues, and run tests. The pre-commit framework (https://pre-commit.com) simplifies managing these hooks. Instead of writing shell scripts, you define hooks in a YAML configuration file, and the framework handles installation and execution. It supports hooks from multiple sources, including popular Python tools like black, flake8, isort, mypy, and many more. By integrating pre-commit hooks into your workflow, you ensure that every commit meets your project's standards, reducing the burden on code reviews and CI pipelines.

# Example of a simple pre-commit hook script (not using framework)
#!/usr/bin/env python
import sys
import subprocess

# Run flake8 on staged files
result = subprocess.run(['flake8', '--max-line-length=88'], capture_output=True, text=True)
if result.returncode != 0:
    print(result.stdout)
    sys.exit(1)
else:
    print("No linting errors found.")
    sys.exit(0)

Setting Up Pre-commit Hooks

To get started, you need to install the pre-commit framework. It's a Python package, so you can install it via pip. It's recommended to install it in your virtual environment or globally. Once installed, you create a .pre-commit-config.yaml file in the root of your Git repository. This file defines the hooks you want to run. The basic structure includes a repos list, where each repo specifies a source (usually a Git repository) and hooks. For Python projects, common repos include https://github.com/psf/black for Black, https://github.com/PyCQA/flake8 for Flake8, and https://github.com/pre-commit/mirrors-mypy for Mypy. After creating the config, run pre-commit install to install the hook scripts into your .git/hooks/ directory. Now, every time you run git commit, the hooks will execute. You can also run all hooks on all files with pre-commit run --all-files.

repos:
  - repo: https://github.com/psf/black
    rev: 23.12.1
    hooks:
      - id: black
        language_version: python3.11
  - repo: https://github.com/PyCQA/flake8
    rev: 6.1.0
    hooks:
      - id: flake8
        args: ['--max-line-length=88']
  - repo: https://github.com/pre-commit/mirrors-mypy
    rev: v1.8.0
    hooks:
      - id: mypy
        additional_dependencies: ['types-requests']
  - repo: https://github.com/pycqa/isort
    rev: 5.13.2
    hooks:
      - id: isort
        args: ['--profile', 'black']

Essential Python Hooks for Code Quality

Several hooks are indispensable for Python projects. Black is an opinionated code formatter that ensures consistent style. Flake8 checks for PEP 8 violations and common errors. isort sorts imports alphabetically and separates them into sections. Mypy performs static type checking, catching type errors before runtime. pylint is another linter that goes beyond PEP 8. For security, bandit finds common security issues. For testing, you can run pytest with a custom hook. Additionally, check-ast validates that files parse as valid Python. You can also include hooks for YAML, JSON, and other file types. The key is to choose hooks that match your project's needs and enforce them consistently.

repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.5.0
    hooks:
      - id: check-ast
      - id: check-yaml
      - id: end-of-file-fixer
      - id: trailing-whitespace-fixer
  - repo: https://github.com/PyCQA/bandit
    rev: 1.7.6
    hooks:
      - id: bandit
        args: ['-s', 'B101']  # Skip assert statements

Customizing Hook Behavior with Arguments

Many hooks accept arguments to customize their behavior. For example, Black supports --line-length to change the maximum line length. Flake8 can be configured with --max-line-length and --ignore to suppress specific errors. isort can use --profile black to align with Black's formatting. Mypy can be given --strict for stricter checks. You can also set environment variables or use configuration files like pyproject.toml or setup.cfg. The args key in the hook configuration allows you to pass command-line arguments. For complex configurations, it's better to use a dedicated config file and reference it.

repos:
  - repo: https://github.com/psf/black
    rev: 23.12.1
    hooks:
      - id: black
        args: ['--line-length=100']
  - repo: https://github.com/PyCQA/flake8
    rev: 6.1.0
    hooks:
      - id: flake8
        args: ['--max-line-length=100', '--ignore=E203,W503']
  - repo: https://github.com/pycqa/isort
    rev: 5.13.2
    hooks:
      - id: isort
        args: ['--profile', 'black', '--line-length=100']

Running Hooks on Specific Files or Stages

By default, pre-commit hooks run on all staged files. You can restrict hooks to specific file patterns using the files and exclude keys. For example, you might want to run a linter only on Python files or exclude generated files. You can also control when hooks run using the stages key. Stages include commit, push, merge-commit, etc. This is useful for running expensive hooks only during push, or running tests only on push. Additionally, you can use always_run: true to run a hook even if no files match. This is useful for hooks that check the entire repository, like a license check.

repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.5.0
    hooks:
      - id: check-ast
      - id: check-yaml
        files: \.yaml$
  - repo: https://github.com/psf/black
    rev: 23.12.1
    hooks:
      - id: black
        stages: [commit]
  - repo: https://github.com/pre-commit/mirrors-mypy
    rev: v1.8.0
    hooks:
      - id: mypy
        stages: [push]
        exclude: ^tests/

Debugging Pre-commit Hook Failures

Sometimes pre-commit hooks fail unexpectedly. Common issues include missing dependencies, incorrect configuration, or version mismatches. The first step is to run pre-commit run --all-files --verbose to see detailed output. If a hook fails due to a missing module, ensure the hook's additional_dependencies are specified. For example, mypy needs types-* packages for third-party libraries. If a hook modifies files (like Black or isort), you may need to stage the changes before committing. Use git diff to see what changed. If a hook is too slow, consider using fail_fast: true or moving it to a later stage. You can also skip hooks temporarily with git commit --no-verify, but use this sparingly.

# Run pre-commit on all files with verbose output
pre-commit run --all-files --verbose

# Check what files are staged
git diff --cached --name-only

# If a hook auto-fixes, stage the changes and recommit
git add -u
git commit -m "Apply pre-commit fixes"

Integrating Pre-commit with CI/CD

Pre-commit hooks are not just for local development; they should also run in CI/CD pipelines to ensure consistency. Most CI systems can run pre-commit run --all-files as a step. This catches issues that might have been skipped locally (e.g., if a developer used --no-verify). You can also use the pre-commit.ci service, which automatically runs hooks on pull requests. To set it up, add a .pre-commit-config.yaml and enable the service. It will comment on PRs with hook failures. This provides a safety net and enforces standards across the team.

name: Pre-commit

on: [push, pull_request]

jobs:
  pre-commit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - uses: pre-commit/action@v3.0.0