Shell Scripting Advanced — SIGTERM Trap Leaves Temp Files

Every DevOps engineer has hit the same wall: a shell script that works beautifully on a laptop but silently corrupts data in production, leaves zombie processes behind after a Kubernetes pod restart, or races itself when two cron jobs fire at the same millisecond. That wall is not a Bash limitation — it's the gap between scripting and engineering. The difference is understanding what the shell is actually doing beneath the syntax.

Shell scripts fail in production for three predictable reasons: they don't handle signals (so cleanup never runs when a container dies), they mismanage file descriptors (so logs get garbled or pipes deadlock), and they make assumptions about subshell variable scope (so a loop that 'obviously' increments a counter does nothing). These aren't beginner mistakes — senior engineers hit them too, because they only surface under specific timing conditions or OS configurations.

By the end of this article you'll be able to write scripts that trap and handle SIGTERM gracefully, use process substitution to diff two live command outputs without temp files, manage file descriptors explicitly to prevent descriptor leaks, implement advisory locking to prevent concurrent runs, and structure a production-grade script with a proper exit framework. These are the patterns that make the difference between a script you trust at 3 AM and one you babysit.

What Is Advanced Shell Scripting?

Advanced shell scripting is the practice of using Bash internals — process substitution, signal traps, subshells, and explicit file descriptor management — to write scripts that survive production conditions. It's not about memorising arcane syntax; it's about understanding how the shell manages processes, file descriptors, and signals under the hood.

You advance from scripting to engineering when you stop treating the shell as a black box. You understand that < <(cmd) is syntactic sugar for a pipe and a file descriptor, that trap cleanup EXIT guarantees cleanup even when your orchestrator sends SIGTERM, and that a pipeline's while loop runs in a subshell — so variables set inside it vanish.

This section sets the stage for the deep dives ahead. You'll see a production-ready script that uses all four techniques right now.

#!/usr/bin/env bash
# TheCodeForge – advanced shell scripting skeleton
set -euo pipefail

cleanup() {
    local rc=$?
    rm -rf /tmp/thecodeforge_*
    exit "$rc"
}
trap cleanup EXIT SIGTERM SIGINT

log_info() {
    echo "[$(date +'%Y-%m-%dT%H:%M:%S')] [INFO] $*" >> /var/log/thecodeforge.log
}

# Process substitution example: compare file listings
diff <(ls /app/current) <(ls /app/backup) > /dev/null || log_info "Differences found"

echo "Script finished normally"

Process Substitution: How It Works and Where It Breaks

Process substitution (<(command)) lets you pass the output of a command as if it were a file. It's syntactic sugar for a temporary named pipe managed by the shell. Use it when you need to feed the result of a command into something that expects a file argument — like diff, comm, or paste.

The shell creates a file descriptor backed by a pipe, then substitutes the path /dev/fd/N in the command line. The command reads from that descriptor. No temp file hits disk, no separate process to manage.

But process substitution only works in bash, zsh, and ksh — not in dash or sh. Portable scripts must fall back to temp files or explicit pipes. Also, each substitution forks a child process. Heavy use can exhaust process limits.

#!/usr/bin/env bash
# TheCodeForge – process substitution with diff
# Compare package lists on two servers via SSH
server_a="web01.prod"
server_b="web02.prod"

diff <(ssh "$server_a" dpkg -l) <(ssh "$server_b" dpkg -l) | head -20

echo "Only on $server_a:"
comm -23 <(ssh "$server_a" dpkg -l | sort) <(ssh "$server_b" dpkg -l | sort)

Signal Traps: The Cleanup That Never Runs

trap registers a command or function to run when the shell receives a signal or exits. The most common production mistake is trapping only SIGINT and forgetting SIGTERM. Kubernetes, Docker, and systemd all send SIGTERM by default when they want a process to stop gracefully.

An even subtler trap: trapping a signal inside a function scope. The trap is global — once set, it applies to the entire shell session. If multiple scripts source a shared library, traps can collide. Always reset traps at the start of your function and restore them afterward.

Another trap: using wait inside a trap handler. Signals are blocked while the trap runs, so wait may return immediately without waiting for children. Use a polling loop instead.

#!/usr/bin/env bash
# TheCodeForge – production-grade trap handler
cleanup() {
    local exit_code=$?
    echo "[$(date)] Cleanup: removing lock file and temp workspace" >&2
    rm -f /var/run/myapp.lock
    rm -rf /tmp/myapp_*
    exit "$exit_code"
}

# Trap EXIT guarantees cleanup on any exit, including SIGTERM
# Also trap the signals explicitly to prevent default termination
trap cleanup EXIT SIGTERM SIGINT

echo "Starting batch job..."
# Simulate work
for i in {1..10}; do
    echo "Processing chunk $i"
    sleep 1
done
echo "Job complete"

Subshells and Variable Scope: The Silent Counter Bug

A subshell is a child shell process spawned with (command) or implicitly by pipelines, command substitution, and background jobs. Variables set inside a subshell are invisible to the parent. This is the root cause of the classic "counter doesn't increment" bug.

Pipelines like seq 10 | while read n; do ((count++)); done run the while loop in a subshell. The variable count is updated in the subshell, then lost when the pipeline completes. The fix: use shopt -s lastpipe (bash 4.2+) to run the last pipeline command in the current shell, or avoid pipelines entirely with process substitution or temporary storage.

Another common pattern: cd inside a subshell doesn't change the parent's directory — great for temporary operations, but surprising if you expect persistent changes.

#!/usr/bin/env bash
# TheCodeForge – subshell variable scope demo
# This DOES NOT work – counter is lost
count=0
seq 5 | while read n; do ((count+=n)); done
echo "Using pipe: count=$count"  # prints 0

# This works – lastpipe
shopt -s lastpipe 2>/dev/null || true
count=0
seq 5 | while read n; do ((count+=n)); done
echo "With lastpipe: count=$count"  # prints 15

# Alternative using process substitution
count=0
while read n; do ((count+=n)); done < <(seq 5)
echo "Process sub: count=$count"  # also 15

File Descriptor Management: Closing Leaks Before They Close You

Every opened file consumes a file descriptor (fd). The shell inherits three: 0 (stdin), 1 (stdout), 2 (stderr). Opening additional fds with exec N>file or exec N<file adds to the per-process limit. A leaked fd means the file cannot be deleted or unmounted, and eventually the process hits the kernel limit (ulimit -n).

In production scripts, fd leaks often occur when: (1) a function opens a descriptor but doesn't close it on error; (2) a trap handler tries to write to a closed fd; (3) multiple scripts share a lock file and the fd is never released.

The safe pattern: wrap fds in a subshell so they close automatically when the subshell exits. Or use explicit exec N>&- to close. For lock files, use flock which manages the fd lifecycle.

#!/usr/bin/env bash
# TheCodeForge – safe fd management
# Using subshell to auto-close fd
(
    exec 3> /var/log/myapp.log
    echo "Logging to fd 3" >&3
    # fd 3 closed when subshell exits
)

# Using explicit close with error handling
myfunction() {
    exec 4< /etc/config/app.conf
    # read config
    exec 4<&-  # close on success
}

# Lock file with flock – fd managed by lock
(
    flock -n 9 || exit 1
    # critical section
    sleep 10
) 9>/var/lock/myapp.lock

Production Patterns: Exit Handlers, Locks, and Logging

Production shell scripts need four things to survive at 3 AM: (1) a guaranteed cleanup function attached to EXIT, (2) advisory locking to prevent concurrent runs, (3) structured logging with timestamps and severity levels, and (4) strict error handling with set -euo pipefail.

The exit handler should remove temp files, release locks, and flush buffers. Locking with flock (not mkdir-based locks) is atomic and works across NFS. Logging should write to a file with timestamps and source context, and rotate via logrotate — never let the script manage rotation.

Error handling: set -e makes the script exit on any unchecked failure. set -u treats unset variables as errors. set -o pipefail ensures pipeline failures propagate. Combine them at the top of every script.

#!/usr/bin/env bash
# TheCodeForge – production script template
set -euo pipefail

# Logging
log_info() { echo "[$(date +'%Y-%m-%dT%H:%M:%S')] [INFO] $*" >> /var/log/myapp.log; }
log_error() { echo "[$(date +'%Y-%m-%dT%H:%M:%S')] [ERROR] $*" >&2; }

# Exit handler
cleanup() {
    local rc=$?
    rm -rf /tmp/myapp_*
    flock -u 9 2>/dev/null || true
    log_info "Cleanup complete, exit code $rc"
    exit "$rc"
}
trap cleanup EXIT

# Advisory lock
(
    flock -n 9 || {
        log_error "Another instance is running"
        exit 1
    }
    log_info "Lock acquired, starting job"
    # actual work here
) 9>/var/lock/myapp.lock

exit 0