Python File Handling — 'w' Mode Truncates on Open

File handling in Python is the bridge between your runtime logic and persistent data. Without it, every variable you compute vanishes the second the script ends. Most developers learn open(), read(), and write() by rote, then ship code that silently corrupts files, hangs on large payloads, or crashes when a CSV lacks a header. Getting file I/O right means understanding modes, context managers, path objects, and the real gotchas that separate a prototype from production-ready code.

What File Handling in Python Actually Does

File handling in Python is the set of operations to read from or write to files on disk, mediated by the built-in open() function. The core mechanic is that open() returns a file object, which acts as a stream between your program and the underlying file descriptor. The mode string you pass — 'r', 'w', 'a', 'x', and their binary variants — determines the initial file pointer position, whether the file is truncated, and whether writes are appended or overwritten.

When you open a file in 'w' mode, Python truncates the file to zero length immediately upon opening — before any write() call. This means the file's existing content is destroyed the moment open() executes, not when you first write data. The file pointer starts at position 0, so every subsequent write overwrites from the beginning. This behavior is consistent across all operating systems because it maps directly to the POSIX open() system call with O_WRONLY | O_CREAT | O_TRUNC flags.

Use 'w' mode when you intend to replace an entire file with new content — for example, writing a fresh configuration file, a log rotation output, or a serialized snapshot. Never use 'w' if you need to preserve existing data or append to a file; that's what 'a' (append) or 'r+' (read-write without truncation) are for. In production systems, accidentally using 'w' instead of 'a' is a common cause of data loss in log aggregators and state dumps.

Opening and Reading Files — and Why the 'with' Statement Exists

Before you can do anything with a file, Python needs a file object — a live connection to that file on disk. You create one with the built-in open() function. The two most important arguments are the file path and the mode: 'r' for read, 'w' for write, 'a' for append, and 'b' tacked on for binary (e.g., 'rb').

Here's the thing most tutorials gloss over: open() acquires an operating-system resource. The OS gives your process a file descriptor — a numbered slot in a limited table. If you open files without closing them, you eventually exhaust that table and get a Too many open files error. Worse, unflushed writes may never reach disk.

The with statement solves this by acting as a guaranteed cleanup mechanism. It calls file.close() the instant the indented block exits — whether that exit is normal, via return, or via a crashing exception. You should use with open(...) every single time, with no exceptions. The only reason the two-line open() / close() pattern still exists in docs is historical; treat it as legacy.

Reading has three flavours: read() loads the entire file into one string (fine for small files, dangerous for large ones), readline() fetches one line at a time, and readlines() returns a list of all lines. For most real work, iterating the file object directly is the cleanest and most memory-efficient approach — Python streams one line at a time without loading everything.

# Scenario: parse a server log file and count how many lines are ERROR level

log_file_path = "server.log"

# Create a sample log file to work with so this script is self-contained
with open(log_file_path, "w", encoding="utf-8") as log_file:
    log_file.write("INFO  2024-06-01 08:00:01 Server started\n")
    log_file.write("INFO  2024-06-01 08:01:15 Request received from 192.168.1.10\n")
    log_file.write("ERROR 2024-06-01 08:01:16 Database connection timeout\n")
    log_file.write("INFO  2024-06-01 08:02:00 Retrying connection\n")
    log_file.write("ERROR 2024-06-01 08:02:05 Max retries exceeded\n")
    log_file.write("INFO  2024-06-01 08:02:06 Falling back to cache\n")

error_count = 0
error_lines = []

# The 'with' block guarantees the file is closed when we're done,
# even if an exception is raised inside the block.
with open(log_file_path, "r", encoding="utf-8") as log_file:
    # Iterating the file object directly reads ONE line at a time —
    # this works correctly even for a 10 GB log file because Python
    # never loads the whole thing into memory at once.
    for line in log_file:
        stripped_line = line.strip()  # remove trailing newline characters
        if stripped_line.startswith("ERROR"):
            error_count += 1
            error_lines.append(stripped_line)

print(f"Total ERROR lines found: {error_count}")
print("\nError details:")
for error in error_lines:
    print(f"  -> {error}")

Writing and Appending — Understanding Why 'w' Mode Is a Trap

Writing to a file sounds simple — and it is, once you understand the critical difference between 'w' (write) and 'a' (append) mode, because confusing them is one of the most common ways to destroy your own data.

Opening a file in 'w' mode does two things: it creates the file if it doesn't exist, and it truncates the file to zero bytes if it does exist — immediately, before you write a single character. That means opening an existing file with 'w' and then crashing before writing anything leaves you with an empty file. The original content is gone.

Append mode ('a') is safer for ongoing data: the file is created if absent, but if it exists, the write cursor starts at the very end. Existing content is untouched. This is exactly what you want for log files, audit trails, or any data you're accumulating over time.

For writing structured data — think generating a report or saving configuration — 'w' is correct because you intentionally want a fresh file each run. For recording events as they happen over time, 'a' is correct. Choosing wrong corrupts data silently, which is why understanding the WHY matters more than memorising the letters.

write() takes a single string. Unlike print(), it does not add a newline automatically — you must include ' ' yourself. writelines() accepts a list of strings and also skips automatic newlines, so each string in your list must already end with ' ' if you want line breaks.

import datetime

activity_log_path = "user_activity.log"

def log_activity(username: str, action: str) -> None:
    """Append a timestamped activity record to the log file.

    We use 'a' mode so that every call adds to the existing log
    rather than overwriting it. This function is safe to call
    from multiple places in a long-running application.
    """
    timestamp = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
    # Build the full log entry as a single string with its own newline
    log_entry = f"[{timestamp}] USER={username} ACTION={action}\n"

    with open(activity_log_path, "a", encoding="utf-8") as activity_log:
        activity_log.write(log_entry)  # 'a' mode: cursor is always at end of file

def generate_daily_report(report_date: str, summary_lines: list[str]) -> None:
    """Write a fresh daily report, replacing any previous report for the same name.

    We intentionally use 'w' mode here because the report is always
    regenerated from scratch — old content should not carry over.
    """
    report_path = f"report_{report_date}.txt"
    header = f"=== Daily Report for {report_date} ===\n\n"

    with open(report_path, "w", encoding="utf-8") as report_file:
        report_file.write(header)
        # writelines() writes each string as-is — no automatic newlines added
        # so we ensure each summary line already ends with '\n'
        report_file.writelines(line + "\n" for line in summary_lines)

    print(f"Report saved to {report_path}")

# Simulate three user events arriving over time
log_activity("alice", "LOGIN")
log_activity("bob", "VIEWED_DASHBOARD")
log_activity("alice", "EXPORTED_CSV")

# Read back the log to confirm all three entries were preserved
print("--- Current activity log ---")
with open(activity_log_path, "r", encoding="utf-8") as activity_log:
    print(activity_log.read())

# Generate a report (uses 'w' mode — intentionally fresh each time)
generate_daily_report(
    report_date="2024-06-01",
    summary_lines=[
        "Total logins: 1",
        "Total page views: 1",
        "Total exports: 1"
    ]
)

Error Handling and File Existence — Writing Code That Doesn't Embarrass You in Production

A file operation in production will eventually fail. The path doesn't exist, the disk is full, the process doesn't have permission, the file is locked by another process. Ignoring this is fine in a throwaway script and a fireable offence in production code.

Python raises specific exceptions for file errors. FileNotFoundError fires when you try to read a file that doesn't exist. PermissionError fires when the OS blocks access. IsADirectoryError fires when you accidentally pass a directory path where a file path was expected. All three are subclasses of OSError, so catching OSError covers all of them — but catching the specific type gives you better error messages.

A common anti-pattern is checking os.path.exists() before opening a file. This looks safe but it's a race condition: between your check and your open(), another process can delete or create that file. The correct pattern is EAFP (Easier to Ask Forgiveness than Permission) — just try to open it and handle the exception. Python was designed around this idiom.

For reading a file that might not exist yet (like a config file on first run), a clean pattern is to catch FileNotFoundError and return a sensible default rather than crashing. For writing, you often want the opposite: verify the destination directory exists and create it if not, using pathlib.Path.mkdir(parents=True, exist_ok=True) before writing.

import json
from pathlib import Path

DEFAULT_CONFIG = {
    "theme": "dark",
    "language": "en",
    "auto_save_interval_seconds": 30
}

CONFIG_PATH = Path("app_data") / "config.json"

def load_config() -> dict:
    """Load config from disk. Returns defaults if the file doesn't exist yet.

    Uses EAFP style: try the operation, handle the specific failure.
    This avoids the race condition that os.path.exists() creates.
    """
    try:
        with open(CONFIG_PATH, "r", encoding="utf-8") as config_file:
            loaded_config = json.load(config_file)
            print(f"Config loaded from {CONFIG_PATH}")
            return loaded_config

    except FileNotFoundError:
        # This is expected on first run — not an error, just a first boot
        print(f"No config file found at {CONFIG_PATH}. Using defaults.")
        return DEFAULT_CONFIG.copy()

    except json.JSONDecodeError as parse_error:
        # The file exists but is malformed — this IS an error worth reporting
        print(f"WARNING: Config file is corrupted ({parse_error}). Using defaults.")
        return DEFAULT_CONFIG.copy()

    except PermissionError:
        # We can't read the file — fail loudly, don't silently use defaults
        raise RuntimeError(
            f"Cannot read config at {CONFIG_PATH}. Check file permissions."
        )

def save_config(config: dict) -> None:
    """Save config to disk, creating the directory structure if it doesn't exist."""
    # mkdir with parents=True creates 'app_data/' if it's missing
    # exist_ok=True means no error if the directory is already there
    CONFIG_PATH.parent.mkdir(parents=True, exist_ok=True)

    with open(CONFIG_PATH, "w", encoding="utf-8") as config_file:
        # indent=2 makes the JSON human-readable — important for config files
        json.dump(config, config_file, indent=2)
        print(f"Config saved to {CONFIG_PATH}")

# First load — file doesn't exist yet
current_config = load_config()
print(f"Theme: {current_config['theme']}")

# User changes a setting
current_config["theme"] = "light"
save_config(current_config)

# Second load — now reads from disk
current_config = load_config()
print(f"Theme after reload: {current_config['theme']}")

Binary Files and pathlib — Handling Images, PDFs and Modern Path Management

Not all files are text. Images, PDFs, audio, compiled code, and serialised data are binary — they contain bytes that aren't valid UTF-8 text. Open them in text mode and you'll get a UnicodeDecodeError at best, or silently corrupted data at worst. Binary mode ('rb', 'wb') tells Python to skip all encoding/decoding and work with raw bytes.

A very common real-world task is copying or processing binary files — resizing images, attaching files to emails, or storing uploaded files from a web form. The pattern is identical to text mode but you read bytes objects instead of strings.

For path manipulation, the modern way in Python 3.4+ is pathlib.Path. Forget string concatenation with os.path.join() — pathlib lets you build paths with / operator, check existence with .exists(), get the file extension with .suffix, list directory contents with .iterdir(), and open files directly via path.open(). It's more readable, cross-platform by default, and object-oriented in a way that makes your intent clear.

When you're processing a directory full of files — a common automation task — pathlib with a generator expression is significantly cleaner than os.listdir() combined with string filtering. The pattern Path('data').glob('*.csv') gives you an iterator of all CSV files in the directory, ready to open.

from pathlib import Path
import shutil

# Scenario: scan a 'downloads' folder and copy image files into an 'images' archive.
# This pattern works identically on Windows, macOS and Linux because pathlib
# handles the slash vs backslash difference for you automatically.

DOWNLOADS_DIR = Path("sample_downloads")
IMAGES_ARCHIVE_DIR = Path("organised") / "images"

# Create sample directory with mixed file types so the script is self-contained
DOWNLOADS_DIR.mkdir(exist_ok=True)
(DOWNLOADS_DIR / "holiday_photo.jpg").write_bytes(b"\xff\xd8\xff" + b"\x00" * 10)
(DOWNLOADS_DIR / "budget.xlsx").write_bytes(b"PK\x03\x04" + b"\x00" * 10)
(DOWNLOADS_DIR / "profile_pic.png").write_bytes(b"\x89PNG" + b"\x00" * 10)
(DOWNLOADS_DIR / "notes.txt").write_text("Some text notes", encoding="utf-8")
(DOWNLOADS_DIR / "logo.jpg").write_bytes(b"\xff\xd8\xff" + b"\x00" * 10)

IMAGES_ARCHIVE_DIR.mkdir(parents=True, exist_ok=True)

image_extensions = {".jpg", ".jpeg", ".png", ".gif", ".webp"}
copy_count = 0

# Path.iterdir() yields Path objects for every item in the directory —
# no string path manipulation needed, no os.path.join() required.
for file_path in DOWNLOADS_DIR.iterdir():
    # .is_file() filters out subdirectories
    # .suffix gives the file extension including the dot, e.g. '.jpg'
    if file_path.is_file() and file_path.suffix.lower() in image_extensions:
        destination = IMAGES_ARCHIVE_DIR / file_path.name

        # shutil.copy2 copies file content AND metadata (timestamps etc.)
        shutil.copy2(file_path, destination)
        copy_count += 1
        print(f"  Copied: {file_path.name} -> {destination}")

print(f"\nDone. {copy_count} image file(s) archived to {IMAGES_ARCHIVE_DIR}")

# Demonstrate reading binary content back
for image_path in IMAGES_ARCHIVE_DIR.glob("*.jpg"):
    # 'rb' mode returns raw bytes — no encoding involved
    with image_path.open("rb") as image_file:
        first_bytes = image_file.read(3)  # JPEG magic bytes are FF D8 FF
        print(f"  {image_path.name} magic bytes: {first_bytes.hex().upper()}")

Working with CSV Files — The Most Common Production File Format

CSV files are everywhere. Exports from databases, spreadsheets, logs, API responses — CSV is the lingua franca of data exchange. But CSV handling has traps: quoting, encoding, newlines inside fields, and missing headers.

Python's csv module handles most of this correctly if you use it right. The common beginner mistake is reading a CSV file manually with for line in file: and splitting on commas — this breaks the moment a field contains a comma or a quoted string. Always use csv.reader or csv.DictReader.

For production, always specify quoting=csv.QUOTE_MINIMAL (the default) for writing, and quoting=csv.QUOTE_NONNUMERIC for reading if all fields should be strings. Use newline='' when opening the file — otherwise the CSV module's newline handling can double up \r on Windows.

Encoding is the biggest silent killer. A CSV file from a French office might be in cp1252 or latin-1. Always pass encoding explicitly, and if you don't know the origin, use encoding='utf-8-sig' to handle the BOM that Excel loves to add.

import csv
from pathlib import Path

# Scenario: read a messy CSV exported from Excel (has BOM, inconsistent quotes)
# and write a clean UTF-8 CSV with standard formatting.

input_path = Path("sales_export.csv")
output_path = Path("sales_cleaned.csv")

# Create a sample 'messy' CSV file for demonstration
with open(input_path, "w", encoding="utf-8-sig") as f:
    f.write("\ufeffProduct,Price,Quantity\r\n")
    f.write('Widget A,"$12.50",3\r\n')
    f.write('Widget B,"$24.99",1\r\n')
    f.write('Widget C,"$5.00",10\r\n')

def clean_price(price_str: str) -> float:
    """Remove currency symbols and whitespace, return float."""
    return float(price_str.replace("$", "").replace(",", "").strip())

# Open with newline='' — critical for csv module portability
with open(input_path, "r", encoding="utf-8-sig", newline='') as infile, \
     open(output_path, "w", encoding="utf-8", newline='') as outfile:

    reader = csv.DictReader(infile)
    fieldnames = reader.fieldnames
    writer = csv.DictWriter(outfile, fieldnames=fieldnames)
    writer.writeheader()

    for row in reader:
        # Clean price field
        row["Price"] = f'{clean_price(row["Price"]):.2f}'
        writer.writerow(row)

print(f"Cleaned CSV written to {output_path}")

# Verify output
with open(output_path, "r", encoding="utf-8") as f:
    print(f.read())

Closing Files Without `with`? You’re Leaking Resources

Manually calling file.close() is the single biggest source of production file bugs I see. If an exception occurs before the close call, the file handle stays open. This silently locks resources — especially on Windows where open files prevent renames or deletes. The with statement guarantees closure even if the block raises. It calls __exit__ which flushes buffers and releases the OS handle. Never open a file without with. It’s not style — it’s correctness. If you must manage lifecycle manually (rare), wrap operations in try/finally and close in the finally block. But with is idiomatic. Use it.

# Bad: leak on exception
f = open("data.txt", "r")
data = f.read()
raise ValueError("simulated crash")
f.close()  # never reached, handle stays open

# Good: guaranteed close
with open("data.txt", "r") as f:
    data = f.read()
    raise ValueError("simulated crash")
# f is closed even after exception

File Existence Checks Are for Amateurs — Use Exception Handling

New devs check if a file exists before reading it. That’s a TOCTOU bug — Time of Check to Time of Use. Between os.path.exists() and open(), the file can be deleted. Instead, open the file and catch FileNotFoundError (or FileExistsError for exclusive creation). This is atomic and safe. Also, never use os.path for paths. Python 3.4+ gives us pathlib which is cleaner and cross-platform. Build paths with Path objects, check with path.exists() if you must, but prefer try/except. Production code is resilient against race conditions. Be resilient.

from pathlib import Path

cfg_path = Path("/etc/app/config.json")

try:
    with open(cfg_path, "r") as f:
        config = f.read()
except FileNotFoundError:
    print(f"Config not found at {cfg_path}. Using defaults.")
    config = "{}"