Shell Scripting Advanced — SIGTERM Trap Leaves Temp Files

Every DevOps engineer has hit the same wall: a shell script that works beautifully on a laptop but silently corrupts data in production, leaves zombie processes behind after a Kubernetes pod restart, or races itself when two cron jobs fire at the same millisecond. That wall is not a Bash limitation — it's the gap between scripting and engineering. The difference is understanding what the shell is actually doing beneath the syntax.

Shell scripts fail in production for three predictable reasons: they don't handle signals (so cleanup never runs when a container dies), they mismanage file descriptors (so logs get garbled or pipes deadlock), and they make assumptions about subshell variable scope (so a loop that 'obviously' increments a counter does nothing). These aren't beginner mistakes — senior engineers hit them too, because they only surface under specific timing conditions or OS configurations.

By the end of this article you'll be able to write scripts that trap and handle SIGTERM gracefully, use process substitution to diff two live command outputs without temp files, manage file descriptors explicitly to prevent descriptor leaks, implement advisory locking to prevent concurrent runs, and structure a production-grade script with a proper exit framework. These are the patterns that make the difference between a script you trust at 3 AM and one you babysit.

What Shell Scripting Advanced Really Means

Advanced shell scripting is the discipline of writing robust, production-grade Bash programs that handle signals, manage resources, and compose complex workflows without leaking state. The core mechanic is explicit control over process lifecycle — trapping SIGTERM, SIGINT, and EXIT to clean up temporary files, release locks, or roll back partial operations. Without these traps, a script killed mid-flight leaves corrupted data and orphaned resources.

In practice, advanced scripting relies on three properties: idempotent cleanup routines, atomic file operations (e.g., mv over write), and strict error handling via set -euo pipefail. A trap on EXIT guarantees cleanup even on unexpected termination, but only if the trap handler itself is idempotent and fast — a slow trap can delay process shutdown and cause cascading failures in orchestrated environments like Kubernetes.

Use advanced patterns when your script manages state beyond its own process — creating temp files, acquiring locks, or modifying shared filesystems. In CI/CD pipelines, cron jobs, or container entrypoints, a missing trap is the difference between a clean retry and a silent corruption that surfaces hours later. This is not about elegance; it's about survival under real-world conditions.

Process Substitution: How It Works and Where It Breaks

Process substitution (<(command)) lets you pass the output of a command as if it were a file. It's syntactic sugar for a temporary named pipe managed by the shell. Use it when you need to feed the result of a command into something that expects a file argument — like diff, comm, or paste.

The shell creates a file descriptor backed by a pipe, then substitutes the path /dev/fd/N in the command line. The command reads from that descriptor. No temp file hits disk, no separate process to manage.

But process substitution only works in bash, zsh, and ksh — not in dash or sh. Portable scripts must fall back to temp files or explicit pipes. Also, each substitution forks a child process. Heavy use can exhaust process limits.

#!/usr/bin/env bash
# TheCodeForge – process substitution with diff
# Compare package lists on two servers via SSH
server_a="web01.prod"
server_b="web02.prod"

diff <(ssh "$server_a" dpkg -l) <(ssh "$server_b" dpkg -l) | head -20

echo "Only on $server_a:"
comm -23 <(ssh "$server_a" dpkg -l | sort) <(ssh "$server_b" dpkg -l | sort)

Signal Traps: The Cleanup That Never Runs

trap registers a command or function to run when the shell receives a signal or exits. The most common production mistake is trapping only SIGINT and forgetting SIGTERM. Kubernetes, Docker, and systemd all send SIGTERM by default when they want a process to stop gracefully.

An even subtler trap: trapping a signal inside a function scope. The trap is global — once set, it applies to the entire shell session. If multiple scripts source a shared library, traps can collide. Always reset traps at the start of your function and restore them afterward.

Another trap: using wait inside a trap handler. Signals are blocked while the trap runs, so wait may return immediately without waiting for children. Use a polling loop instead.

#!/usr/bin/env bash
# TheCodeForge – production-grade trap handler
cleanup() {
    local exit_code=$?
    echo "[$(date)] Cleanup: removing lock file and temp workspace" >&2
    rm -f /var/run/myapp.lock
    rm -rf /tmp/myapp_*
    exit "$exit_code"
}

# Trap EXIT guarantees cleanup on any exit, including SIGTERM
# Also trap the signals explicitly to prevent default termination
trap cleanup EXIT SIGTERM SIGINT

echo "Starting batch job..."
# Simulate work
for i in {1..10}; do
    echo "Processing chunk $i"
    sleep 1
done
echo "Job complete"

Subshells and Variable Scope: The Silent Counter Bug

A subshell is a child shell process spawned with (command) or implicitly by pipelines, command substitution, and background jobs. Variables set inside a subshell are invisible to the parent. This is the root cause of the classic "counter doesn't increment" bug.

Pipelines like seq 10 | while read n; do ((count++)); done run the while loop in a subshell. The variable count is updated in the subshell, then lost when the pipeline completes. The fix: use shopt -s lastpipe (bash 4.2+) to run the last pipeline command in the current shell, or avoid pipelines entirely with process substitution or temporary storage.

Another common pattern: cd inside a subshell doesn't change the parent's directory — great for temporary operations, but surprising if you expect persistent changes.

#!/usr/bin/env bash
# TheCodeForge – subshell variable scope demo
# This DOES NOT work – counter is lost
count=0
seq 5 | while read n; do ((count+=n)); done
echo "Using pipe: count=$count"  # prints 0

# This works – lastpipe
shopt -s lastpipe 2>/dev/null || true
count=0
seq 5 | while read n; do ((count+=n)); done
echo "With lastpipe: count=$count"  # prints 15

# Alternative using process substitution
count=0
while read n; do ((count+=n)); done < <(seq 5)
echo "Process sub: count=$count"  # also 15

File Descriptor Management: Closing Leaks Before They Close You

Every opened file consumes a file descriptor (fd). The shell inherits three: 0 (stdin), 1 (stdout), 2 (stderr). Opening additional fds with exec N>file or exec N<file adds to the per-process limit. A leaked fd means the file cannot be deleted or unmounted, and eventually the process hits the kernel limit (ulimit -n).

In production scripts, fd leaks often occur when: (1) a function opens a descriptor but doesn't close it on error; (2) a trap handler tries to write to a closed fd; (3) multiple scripts share a lock file and the fd is never released.

The safe pattern: wrap fds in a subshell so they close automatically when the subshell exits. Or use explicit exec N>&- to close. For lock files, use flock which manages the fd lifecycle.

#!/usr/bin/env bash
# TheCodeForge – safe fd management
# Using subshell to auto-close fd
(
    exec 3> /var/log/myapp.log
    echo "Logging to fd 3" >&3
    # fd 3 closed when subshell exits
)

# Using explicit close with error handling
myfunction() {
    exec 4< /etc/config/app.conf
    # read config
    exec 4<&-  # close on success
}

# Lock file with flock – fd managed by lock
(
    flock -n 9 || exit 1
    # critical section
    sleep 10
) 9>/var/lock/myapp.lock

Production Patterns: Exit Handlers, Locks, and Logging

Production shell scripts need four things to survive at 3 AM: (1) a guaranteed cleanup function attached to EXIT, (2) advisory locking to prevent concurrent runs, (3) structured logging with timestamps and severity levels, and (4) strict error handling with set -euo pipefail.

The exit handler should remove temp files, release locks, and flush buffers. Locking with flock (not mkdir-based locks) is atomic and works across NFS. Logging should write to a file with timestamps and source context, and rotate via logrotate — never let the script manage rotation.

Error handling: set -e makes the script exit on any unchecked failure. set -u treats unset variables as errors. set -o pipefail ensures pipeline failures propagate. Combine them at the top of every script.

#!/usr/bin/env bash
# TheCodeForge – production script template
set -euo pipefail

# Logging
log_info() { echo "[$(date +'%Y-%m-%dT%H:%M:%S')] [INFO] $*" >> /var/log/myapp.log; }
log_error() { echo "[$(date +'%Y-%m-%dT%H:%M:%S')] [ERROR] $*" >&2; }

# Exit handler
cleanup() {
    local rc=$?
    rm -rf /tmp/myapp_*
    flock -u 9 2>/dev/null || true
    log_info "Cleanup complete, exit code $rc"
    exit "$rc"
}
trap cleanup EXIT

# Advisory lock
(
    flock -n 9 || {
        log_error "Another instance is running"
        exit 1
    }
    log_info "Lock acquired, starting job"
    # actual work here
) 9>/var/lock/myapp.lock

exit 0

Why Your Bash Script Only Worked Once: Execution Environment

You SSH into a box, run your script, and everything works. Five minutes later, it fails for the next person. The problem is never the logic—it's the environment you assumed existed. Every script you write must enforce its own reality. Never trust PATH. Never assume a tool is installed. Never assume the script runs from a specific directory. At minimum, take the absolute path of the script's location, set PATH to known values, and validate required binaries at the top. This is not paranoia. This is what separates a script that runs in production from one that burns a pager at 3 AM. The ten seconds you spend hardening the execution environment saves hours of debugging. Start every advanced script with environment assertions. It is the first line of defense against silent failures that cascade into data loss.

// io.thecodeforge
#!/bin/bash
set -euo pipefail

# Enforce deterministic execution environment
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

# Validate critical dependencies
for cmd in curl jq rsync; do
  if ! command -v "$cmd" &>/dev/null; then
    echo "[FATAL] Missing required command: $cmd" >&2
    exit 1
  fi
done

# Script logic below—safe because we own the sandbox
rsync -avz "${SCRIPT_DIR}/data/" /app/data/

Arrays and Iteration: The Silent Infinite Loop

You wrote a loop over files. It runs forever, fills the disk, and kills the partition. Sound familiar? The classic mistake: using for file in $(ls .log) instead of for file in .log. The first splits on whitespace, performs pathname expansion, and breaks on filenames with spaces or newlines. The second is safe, fast, and correct. The deeper issue is that many developers treat arrays as an afterthought. In production scripting, arrays are essential for safe iteration over unpredictable data: lists of PIDs, file paths, API response fields. Always quote your array expansions: "${array[@]}". This preserves each element as a single token. Combined with set -u, a misspelled variable name becomes a hard error, not silent corruption. Use associative arrays for key-value lookups instead of parsing config files a hundred times. Your loops will be deterministic, your scripts will not eat memory, and your on-call will thank you.

// io.thecodeforge
#!/bin/bash
set -euo pipefail

# Safe iteration: array + quoted expansion
backup_files=(
  "/var/log/app/access.log"
  "/var/log/app/error.log"
)

for file in "${backup_files[@]}"; do
  if [[ ! -f "$file" ]]; then
    echo "[WARN] File not found: $file" >&2
    continue
  fi
  cp "$file" "/backup/$(basename "$file").$(date +%Y%m%d)"
done

# Associative array for config lookup
declare -A SERVICE_PORTS
SERVICE_PORTS["nginx"]=80
SERVICE_PORTS["mysql"]=3306
echo "Nginx port: ${SERVICE_PORTS[nginx]}"