Python Logging: Production Patterns and Best Practices

In production, your code runs without a debugger. When something breaks, you need a detailed record of what happened. That's where logging comes in. Python's built-in logging module is powerful but often misused. Many developers start with print() statements, but that's like using a sticky note when you need a filing cabinet. In this tutorial, you'll learn production-ready logging patterns: how to configure loggers, use structured logging, handle log rotation, and avoid common mistakes. We'll cover real-world scenarios like debugging a silent failure in a web service and tracing requests across microservices. By the end, you'll be able to set up a logging system that helps you sleep better at night.

Why Logging Matters in Production

When your application runs on a server, you can't attach a debugger. Logs are your eyes and ears. They help you understand the flow of execution, detect anomalies, and diagnose failures. Without proper logging, you're flying blind. Consider a web service that occasionally returns 500 errors. Without logs, you have no idea why. With structured logs, you can see the exact request parameters, the stack trace, and the database query that failed. Logging also aids in auditing, compliance, and performance monitoring. In this section, we'll explore the core concepts of Python's logging module: loggers, handlers, formatters, and filters. You'll learn how to set up a basic configuration that scales.

import logging

# Basic configuration
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')

# Create a logger
logger = logging.getLogger(__name__)

logger.info('Application started')
try:
    1 / 0
except ZeroDivisionError:
    logger.exception('Division by zero occurred')

Structured Logging with JSON

Plain text logs are hard to parse. Structured logging outputs logs in JSON format, making it easy for log aggregation tools like ELK, Splunk, or Datadog to index and search. Each log entry becomes a JSON object with fields like timestamp, level, logger, message, and custom context. Python's logging module can be extended with a custom formatter that outputs JSON. Alternatively, you can use libraries like python-json-logger. In this section, we'll implement a JSON formatter from scratch and show how to add extra context (e.g., request ID, user ID) using the extra parameter or a custom filter.

import logging
import json

class JsonFormatter(logging.Formatter):
    def format(self, record):
        log_record = {
            'timestamp': self.formatTime(record, self.datefmt),
            'level': record.levelname,
            'logger': record.name,
            'message': record.getMessage(),
        }
        if hasattr(record, 'extra_fields'):
            log_record.update(record.extra_fields)
        return json.dumps(log_record)

# Setup
handler = logging.StreamHandler()
handler.setFormatter(JsonFormatter())
logger = logging.getLogger('myapp')
logger.addHandler(handler)
logger.setLevel(logging.INFO)

# Log with extra context
logger.info('User login', extra={'extra_fields': {'user_id': 123, 'ip': '192.168.1.1'}})

Log Rotation and Management

Log files grow indefinitely and can fill up your disk. Python's logging.handlers module provides RotatingFileHandler and TimedRotatingFileHandler to manage log files. RotatingFileHandler rotates logs when they reach a certain size, keeping a specified number of backup files. TimedRotatingFileHandler rotates at specific intervals (e.g., daily). In production, you should also consider log compression and cleanup. We'll demonstrate how to configure rotation and discuss best practices like using a separate partition for logs and setting appropriate retention policies.

import logging
from logging.handlers import RotatingFileHandler

handler = RotatingFileHandler('app.log', maxBytes=1024*1024, backupCount=5)
handler.setFormatter(logging.Formatter('%(asctime)s - %(levelname)s - %(message)s'))

logger = logging.getLogger('myapp')
logger.addHandler(handler)
logger.setLevel(logging.INFO)

# Simulate log entries
for i in range(10000):
    logger.info(f'Log entry {i}')

Logging in Multi-Threaded and Async Applications

In multi-threaded applications, logging from multiple threads can cause interleaved messages. Python's logging module is thread-safe, but you need to ensure that your handlers are also thread-safe. For async applications using asyncio, logging can be tricky because the default logging is blocking. You can use a queue-based handler to offload logging to a separate thread. We'll show how to use QueueHandler and QueueListener for non-blocking logging in async apps.

import logging
import asyncio
from logging.handlers import QueueHandler, QueueListener
import queue

# Create a queue and handler
log_queue = queue.Queue()
queue_handler = QueueHandler(log_queue)

# Create a listener that writes to a file
file_handler = logging.FileHandler('async_app.log')
file_handler.setFormatter(logging.Formatter('%(asctime)s - %(levelname)s - %(message)s'))
listener = QueueListener(log_queue, file_handler)
listener.start()

# Configure root logger
root_logger = logging.getLogger()
root_logger.addHandler(queue_handler)
root_logger.setLevel(logging.INFO)

async def task(name):
    logging.info(f'Task {name} started')
    await asyncio.sleep(1)
    logging.info(f'Task {name} finished')

async def main():
    await asyncio.gather(task('A'), task('B'))

asyncio.run(main())
listener.stop()

Centralized Logging and Monitoring

In a distributed system, logs are scattered across multiple servers. Centralized logging aggregates logs into a single platform for search, alerting, and visualization. Common solutions include the ELK stack (Elasticsearch, Logstash, Kibana), Graylog, and cloud services like AWS CloudWatch or Datadog. Python applications can send logs directly to these systems using handlers like logstash.LogstashHandler or watchtower for CloudWatch. We'll show how to send structured logs to a remote syslog server and discuss best practices for log shipping.

import logging
from logging.handlers import SysLogHandler

# Syslog handler (UDP to localhost)
handler = SysLogHandler(address=('localhost', 514), facility=SysLogHandler.LOG_USER)
handler.setFormatter(logging.Formatter('%(name)s: %(message)s'))

logger = logging.getLogger('myapp')
logger.addHandler(handler)
logger.setLevel(logging.INFO)

logger.info('Application started')

Best Practices and Common Pitfalls

In this final section, we summarize best practices: use appropriate log levels, avoid logging sensitive data, log at the right granularity, and use lazy evaluation for expensive log messages. Common pitfalls include logging in tight loops, using print() instead of logging, and forgetting to configure logging in library code. We'll also discuss how to use logging.getLogger(__name__) to create hierarchical loggers and how to configure logging via a dictionary for complex applications.

import logging
import logging.config

# Dictionary configuration
LOGGING_CONFIG = {
    'version': 1,
    'formatters': {
        'default': {
            'format': '%(asctime)s - %(name)s - %(levelname)s - %(message)s',
        },
    },
    'handlers': {
        'console': {
            'class': 'logging.StreamHandler',
            'formatter': 'default',
        },
    },
    'root': {
        'level': 'INFO',
        'handlers': ['console'],
    },
}

logging.config.dictConfig(LOGGING_CONFIG)

logger = logging.getLogger(__name__)

# Lazy evaluation: use %s formatting
logger.debug('User %s logged in', user_id)  # Good: only evaluated if DEBUG enabled
# Avoid: logger.debug(f'User {user_id} logged in')  # Always evaluated