NumPy savetxt — Float64 Truncated by Default '%g'

NumPy's savetxt default fmt='%g' truncates float64 values to ~6 significant digits, silently corrupting data when you round-trip through text files. This caused a model's accuracy to drop from 94% to 61% in one production pipeline — the precision loss from saving feature arrays as CSV propagated into training. The fix is a one-character change to the format string, but understanding when to use text vs. binary I/O prevents this class of bug entirely.

What NumPy savetxt Actually Does to Your Float64 Data

NumPy's savetxt writes array data to a text file using a format string that defaults to '%g'. This format truncates float64 values to approximately 6 significant decimal digits, silently discarding precision. The core mechanic: savetxt converts each element to a string via the format specifier, then writes rows separated by a delimiter. It is the inverse of loadtxt, which parses text back into arrays.

By default, '%g' uses Python's general format: it switches between fixed-point and scientific notation based on magnitude, but caps precision at 6 digits. A float64 can represent about 15–17 significant digits. This mismatch means that round-tripping data through savetxt with default settings introduces errors up to 1e-6 relative, which compounds in iterative algorithms or when comparing results.

Use savetxt when you need human-readable output or interoperability with non-NumPy tools (e.g., CSV for Excel). But never rely on it for lossless storage of float64 data. For precision-critical pipelines — financial calculations, sensor calibration, simulation checkpoints — use NumPy's binary .npy format or HDF5. The default '%g' is a trap for the unwary.

loadtxt and savetxt — Text Files

np.loadtxt() and np.savetxt() are the entry point most people find first, and for good reason — they work with plain text files that any editor, spreadsheet, or downstream tool can open. But they come with constraints that matter the moment you move beyond toy examples.

np.loadtxt() reads the entire file into a single ndarray. Every row must have the same number of columns, and the entire file must be a single dtype — if you mix strings and numbers in the same CSV, it will throw a ValueError. It also reads everything into memory at once, which means a 10GB file needs 10GB of RAM available before you get a single array out.

np.savetxt() writes an ndarray to a text file. The fmt parameter controls how each number is formatted — and this is where the silent precision bug lives. The default '%g' format writes at most 6 significant digits, which is fine for displaying numbers but genuinely lossy for float64 values with more precision than that. If you're using np.savetxt for anything that feeds back into a numerical pipeline, switch fmt to '%.18e', which preserves the full float64 round-trip.

For files that are intended for human inspection — a small sample of predictions, a validation output you're sharing with a stakeholder — CSV is the right choice. For data exchange between pipeline stages, it almost never is.

import numpy as np

# ── Basic save/load round-trip ───────────────────────────────────────────────
data = np.array([[1.0, 2.0, 3.0],
                 [4.0, 5.0, 6.0]])

# Save with header and two decimal places
np.savetxt('data.csv', data, delimiter=',', header='a,b,c', fmt='%.2f')

# Load — skiprows=1 skips the header line
loaded = np.loadtxt('data.csv', delimiter=',', skiprows=1)
print(loaded)
# [[1. 2. 3.]
#  [4. 5. 6.]]

# ── Load only specific columns ───────────────────────────────────────────────
# usecols accepts a single index or a tuple of indices
col_a = np.loadtxt('data.csv', delimiter=',', skiprows=1, usecols=0)
print(col_a)  # [1. 4.]

cols_ab = np.loadtxt('data.csv', delimiter=',', skiprows=1, usecols=(0, 1))
print(cols_ab)
# [[1. 2.]
#  [4. 5.]]

# ── Precision comparison: '%g' vs '%.18e' ────────────────────────────────────
small_value = np.array([0.000001234567890123456])

np.savetxt('lossy.csv', small_value, fmt='%g')
np.savetxt('lossless.csv', small_value, fmt='%.18e')

lossy = np.loadtxt('lossy.csv')
lossless = np.loadtxt('lossless.csv')

print(f'Original:  {small_value[0]:.20f}')
print(f'Lossy %%g: {lossy[0]:.20f}')
print(f'Lossless:  {lossless[0]:.20f}')
print(f'Lossy matches original:    {np.allclose(small_value, lossy, rtol=1e-15)}')
print(f'Lossless matches original: {np.allclose(small_value, lossless, rtol=1e-15)}')

save and load — Fast Binary Format

np.save() and np.load() use NumPy's native binary format, .npy. The format stores three things: a magic number identifying it as NumPy data, a header containing the dtype, shape, and memory order, and the raw bytes of the array exactly as they exist in memory. There is no string conversion, no parsing, no rounding — what goes in comes back out identically.

The performance difference versus CSV is substantial enough to matter in real pipelines. A 1GB float64 array saves in roughly one second with np.save() — the bottleneck is disk write speed. The same array with np.savetxt() takes 30 to 60 seconds because every number has to be converted to its string representation. On load, the difference is similar. If your pipeline is spending meaningful time reading and writing feature matrices as CSV files, switching to .npy will make that time essentially disappear.

File size follows the same pattern. A .npy file for a float64 array is exactly 8 bytes per element plus a small header — there's no overhead for decimal points, separators, or newlines. A CSV for the same data is typically two to three times larger depending on the magnitude of the values.

np.savez() extends this to multiple arrays. It creates a .npz archive — which is structurally a zip file — where each array is stored under a key you provide. You load the whole archive with np.load() and access individual arrays by name. np.savez_compressed() applies zlib compression on top of that, which reduces file size further at the cost of CPU time on both save and load. Compression is worth it for archiving or transferring data; it's usually not worth it for data that gets loaded repeatedly during training.

import numpy as np
import time

# ── Single array: save and load ──────────────────────────────────────────────
arr = np.random.randn(1000, 100).astype(np.float64)

np.save('array.npy', arr)
loaded = np.load('array.npy')

print(loaded.shape)              # (1000, 100)
print(loaded.dtype)              # float64
print(np.allclose(arr, loaded))  # True — exact round-trip

# ── Multiple arrays bundled into one .npz file ───────────────────────────────
X = np.random.randn(10000, 128).astype(np.float32)  # feature matrix
y = np.random.randint(0, 10, size=10000)             # integer labels
metadata = np.array([128, 10], dtype=np.int32)      # shape metadata

np.savez('dataset.npz', features=X, labels=y, meta=metadata)

bundle = np.load('dataset.npz')
print(list(bundle.keys()))            # ['features', 'labels', 'meta']
print(bundle['features'].shape)       # (10000, 128)
print(bundle['labels'].dtype)         # int64
print(bundle['meta'])                 # [128, 10]

# ── Compressed archive for long-term storage ─────────────────────────────────
np.savez_compressed('dataset_compressed.npz', features=X, labels=y)

# ── Rough speed comparison: .npy vs CSV for a 100k x 50 float64 array ────────
large = np.random.randn(100_000, 50)

t0 = time.perf_counter()
np.save('large.npy', large)
t1 = time.perf_counter()
np.savetxt('large.csv', large, fmt='%.18e', delimiter=',')
t2 = time.perf_counter()

print(f'np.save:    {t1 - t0:.3f}s')
print(f'np.savetxt: {t2 - t1:.3f}s')

genfromtxt — Handling Messy Real-World CSVs

Real-world CSV files are not clean. They have missing values where a sensor didn't record, mixed types because someone exported a database table, comment lines at the top explaining the schema, and inconsistent formatting because the file went through three different tools before it reached you. np.loadtxt() handles none of this — it throws a ValueError the moment it encounters something it can't parse.

np.genfromtxt() is the answer for that middle ground: data that's numerical in intent but imperfect in practice. The key behavioural difference is that genfromtxt fills missing values with a placeholder — NaN by default for floats — rather than aborting. It also supports structured dtypes, which let you mix integer columns with float columns and string columns in the same file, and names=True which reads the header row and gives you column access by name.

The names=True output is a structured array, not a regular ndarray. It behaves differently from what most people expect — operations like data['score'] work, but standard array arithmetic does not apply across the whole structure the way it does with a 2D ndarray. It's a lightweight alternative to a Pandas DataFrame for cases where you need named columns but don't want the Pandas dependency.

For genuinely messy data with millions of rows, complex types, date columns, or anything you'll need to reshape and query heavily, Pandas read_csv is the better tool. genfromtxt sits between np.loadtxt and Pandas: it handles imperfect files, but it's not a DataFrame.

import numpy as np
from io import StringIO

# ── Missing values and mixed types ───────────────────────────────────────────
# This is the kind of CSV that arrives from a data export or sensor logger
csv_data = '''id,score,label
1,0.85,cat
2,,dog
3,0.92,cat
4,0.78,
5,,
'''

# genfromtxt fills missing values with NaN — loadtxt would throw here
data = np.genfromtxt(
    StringIO(csv_data),
    delimiter=',',
    skip_header=1,
    dtype=[('id', 'i4'), ('score', 'f8'), ('label', 'U10')],
    missing_values='',
    filling_values={1: np.nan, 2: 'unknown'}  # per-column fill values
)

print(data['id'])     # [1 2 3 4 5]
print(data['score'])  # [0.85 nan  0.92 0.78  nan]
print(data['label'])  # ['cat' 'dog' 'cat' 'unknown' 'unknown']

# ── names=True: auto-parse headers, get column access by name ─────────────────
data2 = np.genfromtxt(
    StringIO(csv_data),
    delimiter=',',
    names=True,
    dtype=None,
    encoding='utf-8'
)

print(data2.dtype.names)   # ('id', 'score', 'label')
print(data2['score'])      # [0.85  nan  0.92  0.78  nan]

# ── Filtering out NaN rows before analysis ───────────────────────────────────
valid_mask = ~np.isnan(data2['score'])
valid_scores = data2['score'][valid_mask]
print(f'Mean score (valid rows only): {valid_scores.mean():.4f}')  # 0.8625

# ── Comment lines — genfromtxt can skip them automatically ───────────────────
csv_with_comments = '''# Exported from sensor array v2.1
# Units: voltage (V)
0.001, 0.002, 0.003
0.004, 0.005, 0.006
'''

clean = np.genfromtxt(
    StringIO(csv_with_comments),
    delimiter=',',
    comments='#'
)
print(clean)
# [[0.001 0.002 0.003]
#  [0.004 0.005 0.006]]

Syntax, Parameters, and the Silent Footguns

Every dev skims syntax. Then they ship a file that breaks in prod at 3 AM. Here's what matters.

savetxt signature: numpy.savetxt(fname, X, fmt='%.18e', delimiter=' ', newline=' ', header='', footer='', comments='# ', encoding=None)

fname is your output path. If it ends with .gz, NumPy transparently gzips — great for log dumps. X is your array (1D or 2D). Got a 3D tensor? It'll silently flatten it along the last axis — no warning. That's bitten teams in latency-critical pipelines.

fmt defaults to '%.18e' which prints every double to 18 decimal places in scientific notation. That's overkill 99% of the time. Use '%.6f' for most sensor data unless you genuinely need sub-micron precision. delimiter defaults to space. CSV? Pass ','. Tab? '\t'. Simple.

header and footer write raw strings. comments is the prefix — defaults to '# '. So header='timestamp=2024-04-01' writes # timestamp=2024-04-01. Screw up comments and your header won't be parseable by loadtxt without tweaking comments there too.

encoding defaults to 'latin1'. If you write UTF-8 headers, loading fails silently on older NumPy (<1.14). Always set encoding='utf-8' in modern workflows.

// io.thecodeforge — python tutorial

import numpy as np
import sys

# Simulate a sensor log with timestamps
sensor_vals = np.arange(100.0)
timestamps = np.linspace(0, 10, 100)

# Don't rely on defaults — specify explicitly
np.savetxt(
    'sensor_log.tsv',
    np.column_stack((timestamps, sensor_vals)),
    fmt='%10.4f',
    delimiter='\t',
    header='time_s\tvalue',
    encoding='utf-8'
)

# Verify load works
loaded = np.loadtxt('sensor_log.tsv', delimiter='\t')
print(f'Loaded shape: {loaded.shape}')
print(f'First row: {loaded[0]}')

Example: Saving Multiple Arrays the Right Way

Can't pass more than one array to savetxt. It expects a single 2D array. If you try np.savetxt('out.csv', arr1, arr2), you get a TypeError and a facepalm. Stack them first with np.column_stack or np.vstack.

Got two arrays of the same first dimension? column_stack merges them column-wise. Data and labels? Same stride or broadcast — pad or slice. This isn't academic; it's what happens when you log sensor readings with timestamps, coordinates with measurements, or any multi-source data.

You could write a loop — but that's how you end up with misaligned columns and a 5 AM pager. Stack once, write once. Use fmt tuples for mixed precision: fmt=['%d', '%10.2f'] for integer IDs and floating readings. savetxt applies them column-by-column.

If the arrays are incompatible shapes, savetxt won't magically fix it — it'll reshape or truncate silently. Validate before you write.

// io.thecodeforge — python tutorial

import numpy as np

# Real-world: logging system metrics
machine_ids = np.array([101, 102, 103], dtype=np.int32)
temperatures = np.array([68.5, 72.1, 65.9], dtype=np.float64)
pressures = np.array([1.02, 1.01, 0.99], dtype=np.float64)

# Stack into a single 2D array — column order matters
data = np.column_stack((machine_ids, temperatures, pressures))

# Mixed format: int for ID, float for readings
np.savetxt(
    'machine_metrics.csv',
    data,
    fmt=['%d', '%.2f', '%.3f'],
    delimiter=',',
    header='id,temp_C,press_atm',
    comments='',
    encoding='utf-8'
)

# Confirm
with open('machine_metrics.csv') as f:
    print(f.read())

Why loadtxt Chokes on Headers — and the One-Liner Fix

You've got a clean CSV with a header row. You call numpy.loadtxt and it throws a ValueError about string conversion. Junior devs blame NumPy. You already know the culprit: every column type gets auto-inferred from the first row of actual data, but loadtxt sees a string header and panics.

The fix is trivial once you know the trap: pass skiprows=1. But that only skips the header — it doesn't save you from mixed types, missing values, or trailing commas. That's when you reach for genfromtxt (covered earlier). But for production pipelines where your data is already validated and clean, loadtxt with skiprows is faster by an order of magnitude.

Real play: if your header contains column names you actually need, use names=True in genfromtxt or roll your own dict mapping with skiprows. Never let a header row be the reason your Monday morning batch job burns down.

// io.thecodeforge — python tutorial

import numpy as np

# Sample CSV with header
with open('temps.csv', 'w') as f:
    f.write("city,high,low\n")
    f.write("Austin,95,72\n")
    f.write("Denver,88,55\n")

# This FAILS — string in first row
# data = np.loadtxt('temps.csv', delimiter=',')

# This WORKS — skip the header
data = np.loadtxt('temps.csv', delimiter=',', skiprows=1)
print("Loaded array:")
print(data)
print(f"dtype: {data.dtype}")

savetxt Precision: Why Your 64-Bit Floats Get Truncated at 1e-6

You saved a double-precision array with 16 significant digits. When you load it back, you lose the 7 least significant digits. This isn't a bug — it's the default fmt='%.18e' behavior in savetxt. By default, it writes with 18 digits after the decimal in scientific notation, which fits in a 64-bit float. But if you use fmt='%f' (default 6 decimal places), you're effectively casting to float32 precision.

Why this matters: in production, where your data is sensor readings or financial tick data, truncating to 1e-6 means cumulative error. We saw a machine learning pipeline silently drift because coordinate data got written with fmt='%f' and reloaded — the offset was 0.000001 per row, 1 million rows later you're 1 meter off.

Rule of thumb: never use the default fmt for critical data. Explicitly set fmt='%.15e' or fmt='%.16g' to guarantee lossless round-trip for float64. For integer data, fmt='%d' is fine. For mixed types, you're back to genfromtxt territory — but that's another war story.

// io.thecodeforge — python tutorial

import numpy as np

data = np.array([3.141592653589793, 2.718281828459045])

# Default fmt='%.18e' — lossless
np.savetxt('full_precision.csv', data, delimiter=',')
loaded_full = np.loadtxt('full_precision.csv', delimiter=',')
print(f"Full precision diff: {data - loaded_full}")

# fmt='%f' — truncated, loses ~7 decimal digits
np.savetxt('truncated.csv', data, delimiter=',', fmt='%f')
loaded_trunc = np.loadtxt('truncated.csv', delimiter=',')
print(f"Truncated diff: {data - loaded_trunc}")