Node.js Streams — 4 GB Upload OOM from Buffer Allocation

Every Node.js server relies on Streams and Buffers whether you realize it or not. Serving a 4 GB video file, parsing a multipart upload, piping a database cursor to an HTTP response — all of these are stream operations under the hood. Get the flow control wrong and your server leaks memory, stalls under load, or corrupts binary data in ways that are genuinely nightmarish to debug at 2 AM.

The core problem Streams solve is the mismatch between producer speed and consumer speed. A database might emit rows faster than your HTTP client can receive them. A file system read might outpace a gzip compressor. Without a flow-control mechanism, the fast side buffers everything into memory until something falls over. Node.js Streams address this with backpressure — a built-in signalling protocol between producers and consumers that has been part of the platform since the beginning but is still widely misunderstood.

I've personally diagnosed production incidents where engineers had been running stream-based services for years without realizing backpressure was being silently violated on every request. The services looked fine right up until traffic doubled during a product launch and three nodes fell over within minutes of each other.

By the end of this article you'll understand how the Buffer class maps onto V8 memory outside the garbage collector, what highWaterMark actually controls (it is not a hard limit — most engineers get this wrong), how to build a production-grade Transform stream that handles partial chunks correctly, why pipeline() is almost always safer than manual pipe() wiring, and the five most dangerous mistakes engineers make when dealing with binary data and stream state in production systems that handle real traffic.

What is Node.js Streams and Buffers?

Node.js Streams and Buffers are the two core primitives for handling data that is too large, too slow, or too continuous to process all at once. Streams provide an abstraction over time — data arrives in chunks across an interval rather than as a single blob that you have to hold in memory. Buffers provide an abstraction over space — they hold raw binary data in a fixed-size region of memory that lives outside V8's normal managed heap.

Every meaningful I/O operation in Node.js — file reads, network sockets, HTTP request bodies, child process stdout, database cursors — ultimately flows through these two primitives. You may be using them indirectly through framework abstractions, but they are always there underneath. Understanding their internals is not an academic exercise; it is the difference between a service that scales predictably and one that mysteriously collapses under load while your dashboards show green.

The reason these primitives exist together is that Buffers alone are not a solution to the large-data problem. A Buffer is just a chunk of memory. Streams are the plumbing that controls when chunks get created, processed, and released. Without streams, you allocate one massive Buffer and hold it until you're done. With streams, you process a small Buffer, release it, and process the next one — memory stays bounded regardless of total data size.

const { createReadStream } = require('fs');
const { createGzip } = require('zlib');
const { pipeline } = require('stream/promises');

// Streaming a large log file through gzip compression.
// At no point does the full file content live in memory —
// each chunk is read, compressed, and written before the next arrives.
async function compressFile(source, destination) {
  const src = createReadStream(source, {
    // Explicitly configure chunk size — 64 KB is the default
    // but for log files on spinning disks, larger reads amortize
    // the syscall overhead significantly.
    highWaterMark: 64 * 1024
  });

  const gzip = createGzip({
    level: 6  // balance between speed and compression ratio
  });

  const dest = require('fs').createWriteStream(destination);

  // pipeline() handles backpressure between all three streams
  // and destroys all of them cleanly if any one fails.
  await pipeline(src, gzip, dest);

  console.log(`Compressed ${source} -> ${destination}`);
}

compressFile('/var/log/app.log', '/var/log/app.log.gz');

Buffer Internals: Memory Outside the Heap

Buffers in Node.js are implemented as Uint8Array typed arrays backed by V8's ArrayBuffer. Since Node.js 8, Buffer memory is tracked through V8's allocator and does show up in process.memoryUsage().heapUsed — but the way it shows up, and the way the garbage collector handles it, is meaningfully different from ordinary JS objects.

A Buffer object in JavaScript is a thin wrapper. The actual byte storage lives in native memory managed by V8's allocator, not in the regular JS heap where your strings and objects live. This matters for two reasons. First, fragmentation patterns differ — native memory can fragment in ways that V8's heap compaction does not address. Second, the GC's visibility into this memory is limited — you can have hundreds of megabytes sitting in Buffer storage while your heap metrics look completely normal.

The practical consequence is that process.memoryUsage().rss is the number you need to watch for Buffer-heavy workloads, not heapUsed. RSS is the total resident memory of your process including native allocations. A healthy Node.js service will show rss comfortably above heapUsed. An unhealthy one — with stream buffers or Buffer allocations leaking — will show rss climbing while heapUsed remains flat, which is exactly the pattern that looks innocuous in heap-focused dashboards until something breaks.

On the allocation side, there are two primary paths. Buffer.alloc(n) zeros out the memory before returning — safe, predictable, slightly slower. Buffer.allocUnsafe(n) skips the zeroing — meaningfully faster for large allocations, but the returned memory contains whatever was in those addresses previously. For internal processing where you fully overwrite every byte before reading, allocUnsafe is fine. For anything that crosses a trust boundary — responses to clients, data written to log files, anything that leaves the process — allocUnsafe is a security vulnerability waiting to manifest.

const { Buffer } = require('buffer');

// Demonstrate the RSS vs heapUsed gap that trips up engineers
// who are only watching heap metrics on their dashboards.
function measureBufferOverhead(sizeInBytes) {
  const before = process.memoryUsage();

  // allocUnsafe skips the zero-fill step — faster, but the returned
  // buffer contains whatever bytes were in that memory previously.
  // Calling fill() here simulates a real workload that overwrites
  // every byte before reading — the only safe use of allocUnsafe.
  const buf = Buffer.allocUnsafe(sizeInBytes);
  buf.fill(0x42);

  const after = process.memoryUsage();

  console.log(`Requested:   ${(sizeInBytes / 1024 / 1024).toFixed(1)} MB`);
  console.log(`RSS delta:   ${((after.rss - before.rss) / 1024 / 1024).toFixed(1)} MB`);
  console.log(`Heap delta:  ${((after.heapUsed - before.heapUsed) / 1024 / 1024).toFixed(1)} MB`);
  console.log(`External:    ${((after.external - before.external) / 1024 / 1024).toFixed(1)} MB`);

  // Releasing the reference makes the Buffer eligible for GC
  // but GC is not immediate — RSS may stay elevated until the next collection.
  return null;
}

measureBufferOverhead(100 * 1024 * 1024); // 100 MB

Backpressure: The Flow Control Protocol

Backpressure is the mechanism by which a slow consumer tells a fast producer to stop sending data until it catches up. In Node.js streams, this signal travels through the return value of writable.write(). When write() returns false, it means the writable stream's internal buffer has exceeded its highWaterMark threshold and the producer needs to stop. The producer must wait for the 'drain' event — emitted when the internal buffer has dropped back below the threshold — before writing again.

This is not optional and it is not a suggestion. Ignoring the return value of write() is the single most common source of memory leaks in stream-based Node.js applications. The writable stream will keep accepting data into its internal buffer — it will not throw an error or refuse the write. It will just quietly accumulate memory at whatever rate the producer pushes data. Eventually the process runs out of memory, typically hours after the problematic code path was first exercised.

What makes this especially dangerous is the failure mode. The service does not crash immediately. It degrades slowly. Response times get worse as the GC works harder. Memory alarms trigger at 80%, then 90%. Engineers restart the service and mark the incident as 'transient.' The cycle repeats until someone looks closely at the RSS trend over multiple days and recognizes the pattern.

The highWaterMark option — 16 KB default for objectMode streams, 64 KB for byte streams — sets the threshold at which backpressure is signalled. It is a soft limit, not a hard wall. Data written beyond the threshold is not discarded or rejected. The stream accepts it, queues it internally, and simply signals the producer to slow down. Think of it as a gauge reading, not a circuit breaker.

const { createReadStream, createWriteStream } = require('fs');
const { pipeline } = require('stream/promises');

// Manual backpressure handling — shown here for educational purposes.
// Understanding this is important even if you never write it yourself,
// because pipe() and pipeline() do exactly this under the hood.
function copyWithBackpressure(src, dest) {
  const reader = createReadStream(src);
  const writer = createWriteStream(dest);

  reader.on('data', (chunk) => {
    // write() returns false when the internal buffer exceeds highWaterMark.
    // This is the backpressure signal — not an error, just a flow-control message.
    const canContinue = writer.write(chunk);

    if (!canContinue) {
      // Pause the readable stream — it will stop emitting 'data' events.
      reader.pause();

      // 'drain' fires when the writable buffer drops below highWaterMark.
      // Use once() not on() — drain can fire multiple times per session
      // and we only want to resume once per pause.
      writer.once('drain', () => {
        reader.resume();
      });
    }
  });

  reader.on('end', () => writer.end());

  // Must handle errors on both streams independently.
  // A failure on the writer does not automatically destroy the reader.
  reader.on('error', (err) => {
    console.error('Read error:', err.message);
    writer.destroy(err);
  });

  writer.on('error', (err) => {
    console.error('Write error:', err.message);
    reader.destroy(err);
  });
}

// This is how you should actually write this in production.
// pipeline() does everything copyWithBackpressure() does above,
// plus better error propagation and automatic cleanup on failure.
async function copyWithPipeline(src, dest) {
  await pipeline(
    createReadStream(src),
    createWriteStream(dest)
  );
}

Transform Streams: Data Processing in Flight

A Transform stream sits between a readable source and a writable destination. It receives chunks through its _transform() method, processes them, and pushes results downstream via this.push(). This is the backbone of data pipelines — compression, encryption, CSV parsing, protocol framing, metrics collection, and byte counting all belong in Transform streams.

The _transform(chunk, encoding, callback) method signature is important to understand precisely. The callback signals that this chunk has been fully processed and the stream is ready to receive the next one. It must be called exactly once per _transform() invocation — no more, no less. Calling it zero times hangs the pipeline indefinitely with no error emitted. Calling it more than once corrupts the internal state and typically causes data duplication or a crash.

The _flush(callback) method is called exactly once, after all input has been consumed and before the 'finish' event fires on the writable side. This is where you emit any data that has been buffered across chunk boundaries — the tail of a partial JSON line, the last bytes of a framed message, the final row of a CSV file. If your Transform buffers anything between chunks and does not implement _flush(), you are silently dropping data at the end of every processed stream. This is a very common bug that unit tests almost never catch because test inputs tend to be clean multiples of the chunk size.

Transform streams can operate in byte mode or objectMode. Object mode treats each chunk as a discrete JavaScript value instead of a Buffer. This is useful for structured data pipelines — reading database rows through a Transform that enriches each row, or processing parsed JSON objects through a series of validation and transformation steps. Object mode streams have a default highWaterMark of 16 objects rather than 64 KB.

const { Transform } = require('stream');

// A production-grade JSON Lines parser — the kind you'd actually ship.
//
// Design decisions worth noting:
// 1. Incoming data is binary/string; output is objects (readableObjectMode: true)
// 2. Partial lines that span chunk boundaries are preserved in this._remainder
// 3. Invalid JSON lines emit a custom event rather than halting the pipeline
// 4. _flush() handles the final partial line that never gets a trailing newline
class JsonLinesParser extends Transform {
  constructor(options = {}) {
    super({ ...options, readableObjectMode: true });
    this._remainder = '';
    this._lineCount = 0;
    this._errorCount = 0;
  }

  _transform(chunk, encoding, callback) {
    // Prepend any leftover bytes from the previous chunk,
    // then split on newlines. The last element may be partial.
    const text = this._remainder + chunk.toString('utf8');
    const lines = text.split('\n');

    // pop() removes and returns the last element.
    // If the chunk ended with a complete newline, this will be an empty string.
    // If the chunk ended mid-line, this will be the partial line.
    // Either way, saving it is correct.
    this._remainder = lines.pop();

    for (const line of lines) {
      const trimmed = line.trim();
      if (!trimmed) continue; // skip blank lines between records

      this._lineCount++;

      try {
        const obj = JSON.parse(trimmed);
        this.push(obj); // emit parsed object downstream
      } catch (err) {
        this._errorCount++;
        // Emit metadata about the bad line but continue processing.
        // Halting the pipeline on a single malformed record would be wrong
        // for most log-processing use cases.
        this.emit('invalid_json', {
          line: this._lineCount,
          raw: trimmed,
          error: err.message
        });
      }
    }

    // Signal that this chunk is fully processed.
    // The pipeline will not deliver the next chunk until this is called.
    callback();
  }

  _flush(callback) {
    // Process whatever is left in the remainder buffer.
    // Without _flush(), the final line of every file is silently dropped
    // unless the file ends with a trailing newline — and many do not.
    const remaining = this._remainder.trim();

    if (remaining) {
      this._lineCount++;
      try {
        this.push(JSON.parse(remaining));
      } catch (err) {
        this._errorCount++;
        this.emit('invalid_json', {
          line: this._lineCount,
          raw: remaining,
          error: err.message
        });
      }
    }

    // This log line runs in production on every file — useful for monitoring.
    console.error(`[JsonLinesParser] ${this._lineCount} lines processed, ${this._errorCount} parse errors`);

    // Must call callback to signal that flushing is complete.
    // Forgetting this hangs the pipeline at the very end — often in _flush()
    // is exactly the last place engineers think to look.
    callback();
  }
}

module.exports = { JsonLinesParser };

pipeline() vs pipe(): Why You Should Almost Always Use pipeline()

The .pipe() method connects a readable stream to a writable stream with automatic backpressure handling. It is simple, it works, and it has been in Node.js since the beginning. It is also dangerously incomplete for production use, and I would argue that every usage of raw pipe() in a production codebase should come with a comment explaining why pipeline() was not the right tool.

The critical gap in pipe() is error propagation. When any stream in a pipe() chain emits an 'error' event, only that stream is automatically destroyed. The other streams in the chain keep running — reading, writing, holding file descriptors open, consuming resources. A failed Transform stream leaves the source Readable still emitting data into a destroyed destination. File descriptors stay open. The source stream may continue pulling data from the OS without any consumer to receive it. Under sustained error conditions — a downstream service that is timing out on every request — this pattern exhausts the OS file descriptor limit within hours.

stream.pipeline() (and its Promise-based version in stream/promises) was built to close this gap. It connects all streams in the chain, attaches error handlers to each one, and destroys the entire chain when any participant fails. The Promise-based version composes naturally with async/await — a pipeline failure becomes a rejected Promise that you handle with try/catch like any other async error.

There is exactly one case where I reach for pipe() over pipeline() in 2026: simple REPL-style scripting where I am connecting two streams inline and the total lifespan of the operation is a few seconds. For any service-level code, pipeline() is the default.

const { pipeline } = require('stream/promises');
const { createReadStream, createWriteStream } = require('fs');
const { createGzip } = require('zlib');
const { createCipheriv, randomBytes } = require('crypto');

// A realistic production pipeline: read a log file,
// compress it with gzip, encrypt the result with AES-256-CBC,
// and write to an archive destination.
//
// Four streams in the chain. pipe() would require three
// separate error handler pairs and manual cleanup logic.
// pipeline() handles all of it.
async function archiveLogFile(source, destination, encryptionKey) {
  // Generate a fresh IV for every encryption operation.
  // Reusing IVs with the same key is a cryptographic weakness.
  const iv = randomBytes(16);
  const cipher = createCipheriv('aes-256-cbc', encryptionKey, iv);

  try {
    await pipeline(
      createReadStream(source),
      createGzip({ level: 6 }),
      cipher,
      createWriteStream(destination)
    );

    console.log(`Archived: ${source} -> ${destination}`);

    // Return the IV so the caller can store it for decryption.
    // Without this, the encrypted archive is permanently unreadable.
    return { success: true, iv: iv.toString('hex') };
  } catch (err) {
    // pipeline() rejects with the first error encountered.
    // All four streams have already been destroyed by this point —
    // no fd leak, no lingering background reads, no partial output.
    console.error(`Archive failed: ${err.message}`);
    return { success: false, error: err.message };
  }
}

module.exports = { archiveLogFile };

Production Patterns and Anti-Patterns

After working with stream-based services that process terabytes of data daily across several production systems, certain patterns have proven consistently reliable and others have consistently caused incidents. This section is the distillation of those patterns into rules you can apply immediately.

Pattern 1: Set highWaterMark explicitly and deliberately. The defaults are general-purpose tuning that is not optimized for any specific workload. A streaming video server sending to many simultaneous clients might want a 1 MB highWaterMark to minimize syscall overhead per chunk. A real-time event processor that needs low latency between record arrival and downstream delivery might want 8 KB. A batch ETL pipeline transferring to a fast local disk might want 256 KB. Measure, set it explicitly, document why.

Pattern 2: Always guard pipelines with timeouts. A stream that never completes — because the downstream service is hanging, because a client disconnected without the TCP stack surfacing it, because a file descriptor points to a filesystem that went into a degraded state — holds resources open indefinitely. Wire AbortController to every long-running pipeline and clean up on abort. The timeout should reflect your SLA for the operation, not an arbitrary large number.

Pattern 3: Use objectMode for structured data pipelines. Manually serializing and deserializing between Buffers and structured values inside a byte-mode Transform is error-prone and wasteful. Object mode lets you push and receive plain JavaScript objects, handling the conceptual boundary between binary I/O and application logic at a single, clear point in the pipeline.

Pattern 4: Test your error paths with real stream failures, not mocked ones. The bugs in stream code almost always live in the error and cleanup paths. Create integration tests that inject actual errors — close a file descriptor mid-stream, send malformed data, introduce artificial delays that trigger timeouts. Your happy-path test suite will not find these.

const { pipeline } = require('stream/promises');
const { createReadStream, createWriteStream } = require('fs');
const { Transform } = require('stream');

// Production pattern: timeout-guarded pipeline using AbortController.
//
// This is the pattern I reach for any time a stream operation
// has an SLA that matters — which in practice means almost always.
// The AbortController gives a clean cancellation signal that
// pipeline() understands and propagates to all participants.
async function processWithTimeout(filePath, timeoutMs) {
  const ac = new AbortController();
  const timeout = setTimeout(() => {
    console.error(`[processWithTimeout] Aborting after ${timeoutMs}ms`);
    ac.abort();
  }, timeoutMs);

  // Byte-mode input -> object-mode output:
  // Each line of text becomes a plain object downstream.
  const parser = new Transform({
    readableObjectMode: true,
    transform(chunk, encoding, callback) {
      const lines = chunk.toString('utf8').split('\n');
      for (const line of lines) {
        const trimmed = line.trim();
        if (trimmed) this.push({ data: trimmed, ts: Date.now() });
      }
      callback();
    }
  });

  // Object-mode input -> byte-mode output:
  // Each object is serialized back to a JSON Lines string for writing.
  const serializer = new Transform({
    writableObjectMode: true,
    transform(obj, encoding, callback) {
      this.push(JSON.stringify(obj) + '\n');
      callback();
    }
  });

  try {
    await pipeline(
      createReadStream(filePath, { highWaterMark: 64 * 1024 }),
      parser,
      serializer,
      createWriteStream(filePath + '.processed'),
      { signal: ac.signal } // pipeline respects AbortController natively
    );

    console.log('[processWithTimeout] Processing complete');
  } catch (err) {
    if (err.name === 'AbortError') {
      // Clean cancellation — all streams are already destroyed by pipeline().
      console.error(`[processWithTimeout] Timed out after ${timeoutMs}ms`);
    } else {
      // Something else went wrong — log with full context for debugging.
      console.error(`[processWithTimeout] Failed: ${err.message}`);
      throw err; // re-throw non-timeout errors so callers can handle them
    }
  } finally {
    // Always clear the timeout — if pipeline() completed before it fired,
    // leaving it running delays process exit and clutters event loop metrics.
    clearTimeout(timeout);
  }
}

module.exports = { processWithTimeout };

Four Stream Types and When You Actually Use Each One

The four stream types—Readable, Writable, Duplex, and Transform—aren't academic categories. They map directly to real I/O patterns. Readable streams pull data from a source. Think file reads, HTTP responses, or database cursors. Writable streams push data to a destination: file writes, HTTP requests, or stdout. Duplex streams handle both directions in one object, like a TCP socket where you read incoming bytes and write outgoing bytes simultaneously. Transform streams sit between a readable and writable stream, modifying data in flight. Use them for compression, encryption, or parsing CSV rows without loading the entire file. The common mistake? Using a Duplex when you only need a Transform. Duplex streams force you to manage both sides. If your goal is to modify data, use Transform. It handles backpressure automatically and keeps your pipeline predictable. Pick the right tool: Readable for sources, Writable for sinks, Duplex for bidirectional protocols, Transform for modifications.

const { Transform } = require('stream');
const { createReadStream, createWriteStream } = require('fs');

const uppercaseTransform = new Transform({
  transform(chunk, encoding, callback) {
    this.push(chunk.toString().toUpperCase());
    callback();
  }
});

createReadStream('input.txt')
  .pipe(uppercaseTransform)
  .pipe(createWriteStream('output.txt'));

console.log('Transformed with backpressure control');

Buffers: The Raw Binary Containers You Can't Ignore

Buffers are Node.js's mechanism for handling raw binary data. They live outside the V8 heap, directly in the C++ layer. This is by design: it avoids garbage collection pressure when processing large files or network payloads. Buffers are fixed-size allocations. Once created, resizing means allocating a new buffer and copying data—expensive and avoidable. Always pre-allocate when you know the size. Buffer.from() creates from strings, arrays, or other buffers. Buffer.alloc() creates zero-filled buffers safely. Never use new Buffer()—it's deprecated and can expose uninitialized memory. Buffers interact with streams through chunk events. Each 'data' event passes a Buffer chunk. Converting every chunk to a string with toString('utf-8') is safe but costly. For binary protocols (images, protobuf), keep chunks as buffers and use Buffer.concat() only when you've received all data. Memory is not free. Buffers consume raw memory outside V8's GC. Monitor with process.memoryUsage().bufferUsed.

const fs = require('fs');
const { once } = require('events');

const stream = fs.createReadStream('large-file.bin');
const chunks = [];

stream.on('data', (chunk) => {
  chunks.push(chunk);
  console.log(`Chunk size: ${chunk.length} bytes`);
});

await once(stream, 'end');

const fullBuffer = Buffer.concat(chunks);
console.log(`Total size: ${fullBuffer.length} bytes`);
console.log(`Buffer memory: ${process.memoryUsage().bufferUsed} bytes`);

Backpressure: Why Your Pipeline Breaks Without Flow Control

Backpressure is the mechanism that prevents a fast readable stream from overwhelming a slow writable stream. Without it, your process memory balloons until the system crashes. Node.js handles backpressure automatically for most built-in streams. When your writable stream's internal buffer exceeds highWaterMark (default 16KB), it returns false from write(). The readable stream pauses. Data stops flowing. This is good. The problem? Custom streams and misconfigured transforms can bypass this. Never ignore the return value of write(). Always honor the 'drain' event. In pipe() vs pipeline(), pipeline() handles backpressure correctly—pipe() does not in edge cases. Test your pipelines with a slow sink. Use a Writable stream that waits before calling callback to simulate a slow database. Watch for memory growth. If it climbs, your transform isn't respecting backpressure. The fix: in your _transform method, always call callback() when you're ready for the next chunk. Never queue chunks internally without limits.

const { Writable, Transform, pipeline } = require('stream');

const slowWriter = new Writable({
  highWaterMark: 1024,
  write(chunk, encoding, callback) {
    setTimeout(callback, 10); // simulate slow disk
  }
});

const fastReader = new Transform({
  transform(chunk, encoding, callback) {
    this.push(chunk);
    callback();
  }
});

let bytes = 0;
fastReader.on('data', (chunk) => {
  bytes += chunk.length;
  console.log(`Memory: ${process.memoryUsage().heapUsed} bytes`);
});

pipeline(fastReader, slowWriter, (err) => {
  if (err) console.error(err);
  console.log('Done');
});