Cron Jobs in Linux — Why Your Backup Script Ran Twice

Every production system has a graveyard of tasks that someone used to do manually — rotating log files, sending weekly reports, backing up databases, clearing temp folders. Done by hand, these tasks get forgotten, delayed, or skipped on holidays. Done wrong, they take down services. Cron is the Linux answer to this problem: a built-in scheduler that's been running quietly on Unix systems since 1975, and still powers millions of automated workflows today.

The core problem cron solves is reliability. Humans forget. Cron doesn't. If you need something to happen at a predictable time — daily, hourly, every 15 minutes, or at 3:47am on the last day of the month — cron handles it without a process manager, without a paid SaaS tool, and without a single line of application code. It lives at the OS level, which means it works regardless of what language your app is written in or whether your app is even running.

By the end of this article you'll be able to write cron expressions confidently, manage crontab files without breaking things, debug jobs that silently fail, and apply the real-world patterns that DevOps engineers actually use in production. You'll also know the three mistakes that catch almost everyone the first time they use cron — and exactly how to avoid them.

Why Your Backup Script Ran Twice — The Real Story of Cron Jobs in Linux

A cron job is a scheduled task on Linux, driven by the cron daemon (crond). The core mechanic: crond reads crontab files — one per user — and executes commands at times specified by five fields (minute, hour, day of month, month, day of week). That's it. No retry logic, no overlap protection, no alerting. It's a fire-and-forget timer.

In practice, cron jobs run with the environment of the user who owns the crontab, not a full login shell. That means PATH, variables, and even the working directory are minimal — often just /bin:/usr/bin. A script that works interactively may fail under cron because it can't find commands or expects a different $HOME. Also, if a job takes longer than its interval, cron starts a new instance anyway — no locking, no queuing.

Use cron for periodic, stateless, short-lived tasks: log rotation, temp file cleanup, health pings. Avoid it for jobs that must run exactly once, require guaranteed delivery, or need complex dependency chains. For those, you need a scheduler like systemd timers or a distributed queue. Cron is a hammer — don't use it on screws.

How Cron Actually Works — The Daemon, the Crontab, and the Schedule

Cron is a daemon — a background process that starts when your system boots and never stops. Its name comes from 'chronos', the Greek word for time. Every minute, the cron daemon wakes up, checks all the crontab files on the system, and asks: 'Is there anything I should run right now?' If yes, it fires the job. Then it goes back to sleep until the next minute.

A crontab (cron table) is just a plain text file that lists scheduled jobs. Each line in a crontab is one job: five time fields followed by the command to run. You never edit this file directly — you use the crontab command, which validates the format before saving, protecting you from syntax errors that would silently break everything.

There are two types of crontabs you'll work with. The first is user crontabs — each Linux user has their own, and jobs run as that user. The second is the system crontab at /etc/crontab and files dropped into /etc/cron.d/, which include an extra field specifying which user to run the job as. For most application-level automation, user crontabs are the right choice. For system-level jobs like log rotation, the system crontab is used.

The key mental model: cron doesn't track whether a previous job finished. If your job takes longer than its schedule interval, you can end up with two copies running at the same time. That's one of the most dangerous production gotchas, and we'll cover how to handle it.

# ─────────────────────────────────────────────────
# MANAGING YOUR CRONTAB — the essential commands
# ─────────────────────────────────────────────────

# Open your crontab in the default editor (usually nano or vi)
# This is the ONLY safe way to edit your crontab
crontab -e

# List all your current cron jobs — great for auditing
crontab -l

# Remove ALL your cron jobs (dangerous — no confirmation prompt!)
crontab -r

# Edit the crontab for a specific user (must run as root)
crontab -u deploy_user -e

# ─────────────────────────────────────────────────
# ANATOMY OF A CRON EXPRESSION
# ─────────────────────────────────────────────────
# Each job line follows this exact structure:
#
# ┌──────────── minute        (0 - 59)
# │  ┌─────────── hour          (0 - 23)
# │  │  ┌──────────── day of month  (1 - 31)
# │  │  │  ┌─────────── month         (1 - 12)
# │  │  │  │  ┌──────────── day of week   (0 - 7, both 0 and 7 = Sunday)
# │  │  │  │  │
# *  *  *  *  *  command_to_run

# ─────────────────────────────────────────────────
# REAL EXAMPLES with plain-English explanations
# ─────────────────────────────────────────────────

# Run database backup every day at 2:30am
30 2 * * * /opt/scripts/backup_postgres.sh

# Clear the application temp directory every hour
0 * * * * rm -rf /var/app/tmp/*

# Send a weekly analytics report every Monday at 9am
0 9 * * 1 /opt/scripts/send_weekly_report.py

# Run a health check every 15 minutes
*/15 * * * * /opt/scripts/health_check.sh

# Archive logs on the 1st of every month at midnight
0 0 1 * * /opt/scripts/archive_logs.sh

# Run a job only on weekdays (Mon-Fri) at 6am
0 6 * * 1-5 /opt/scripts/sync_business_data.sh

Writing Production-Ready Cron Jobs — Logging, Environments, and Locking

Here's where most tutorials stop — and where most real problems start. A cron job that runs date will work fine. A cron job that runs your Python script will almost certainly fail silently the first time, and here's why: cron runs with a minimal environment. It doesn't load your .bashrc, .bash_profile, or any of the environment variables you set in your shell session. That means PATH, PYTHONPATH, NODE_ENV, database credentials, API keys — none of it is there unless you explicitly provide it.

The second production concern is logging. By default, cron swallows all output. If your script crashes, you'll never know unless you've set up logging. The fix is simple: redirect both stdout and stderr to a log file on every single cron job.

The third concern is job overlap. If your backup script takes 90 minutes but runs every hour, you'll eventually have two copies fighting over the same files. The solution is a lock file — a mechanism where the script checks if another copy of itself is already running and exits gracefully if so. The flock utility makes this one line.

These three patterns — explicit environment, output logging, and job locking — are what separate a toy cron job from a production one. The example below shows all three working together in a realistic database backup script.

#!/bin/bash
# ─────────────────────────────────────────────────────────────────
# backup_postgres.sh
# Production-ready cron job: PostgreSQL backup with logging + locking
# Scheduled via crontab to run daily at 2:30am:
#   30 2 * * * /bin/bash /opt/scripts/backup_postgres.sh
# ─────────────────────────────────────────────────────────────────

# ── 1. EXPLICIT ENVIRONMENT ──────────────────────────────────────
# Cron's PATH is minimal (/usr/bin:/bin). Set it explicitly so
# commands like pg_dump, aws, gzip are all found correctly.
export PATH="/usr/local/bin:/usr/bin:/bin"

# Load app-specific environment variables from a secure file
# This file contains DB_USER, DB_NAME, S3_BUCKET etc.
# Permissions on this file should be 600 (owner read/write only)
source /etc/app/backup.env

# ── 2. LOGGING SETUP ─────────────────────────────────────────────
LOG_DIR="/var/log/app_backups"
LOG_FILE="${LOG_DIR}/backup_$(date +%Y-%m-%d).log"
TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S')

# Create log directory if it doesn't exist
mkdir -p "$LOG_DIR"

# Redirect ALL output (stdout + stderr) to the log file
# The tee command also prints to the terminal when run manually
exec > >(tee -a "$LOG_FILE") 2>&1

echo "[$TIMESTAMP] ─── Backup job started ───"

# ── 3. JOB LOCKING — prevent overlapping runs ────────────────────
LOCK_FILE="/tmp/backup_postgres.lock"

# flock acquires an exclusive lock on the lock file.
# -n means "non-blocking" — if the lock is already held, exit immediately.
# 9 is the file descriptor we're using for the lock.
exec 9>"$LOCK_FILE"
if ! flock -n 9; then
    echo "[$TIMESTAMP] Another backup is already running. Exiting."
    exit 1
fi

# ── 4. THE ACTUAL WORK ───────────────────────────────────────────
BACKUP_FILENAME="${DB_NAME}_$(date +%Y%m%d_%H%M%S).sql.gz"
BACKUP_PATH="/tmp/${BACKUP_FILENAME}"

echo "[$TIMESTAMP] Dumping database: $DB_NAME"

# pg_dump connects to PostgreSQL and streams SQL to gzip for compression
# PGPASSWORD is read from the sourced env file above
pg_dump -U "$DB_USER" -h "$DB_HOST" "$DB_NAME" | gzip > "$BACKUP_PATH"

# Check if the dump succeeded — never assume it worked
if [ $? -ne 0 ]; then
    echo "[$TIMESTAMP] ERROR: pg_dump failed! Aborting upload."
    exit 1
fi

echo "[$TIMESTAMP] Dump complete. Uploading to S3: s3://${S3_BUCKET}/"

# Upload to S3 — credentials come from the sourced env file
aws s3 cp "$BACKUP_PATH" "s3://${S3_BUCKET}/postgres-backups/${BACKUP_FILENAME}"

if [ $? -eq 0 ]; then
    echo "[$TIMESTAMP] Upload successful. Cleaning up local file."
    rm -f "$BACKUP_PATH"
else
    echo "[$TIMESTAMP] ERROR: S3 upload failed! Local backup retained at $BACKUP_PATH"
    exit 1
fi

echo "[$TIMESTAMP] ─── Backup job completed successfully ───"

Special Schedules, System Crontabs, and When to Use Alternatives

Once you're comfortable with the five-field syntax, there are shorthand strings that make common schedules far more readable. Instead of 0 0 * you can write @daily. These are called cron nicknames and every modern cron daemon supports them. They're self-documenting and much harder to misread.

Beyond user crontabs, Linux ships with a set of system-managed cron directories. Drop an executable script into /etc/cron.daily/ and it will run once a day — no crontab editing required. The actual run times are controlled by the run-parts entries in /etc/crontab. These are perfect for system maintenance tasks packaged by software installers.

That said, cron isn't always the right tool. It has real limitations: it has no dependency management (it can't wait for Job A to finish before starting Job B), it doesn't retry on failure, it has no built-in alerting, and it doesn't scale across multiple machines. For anything more complex — multi-step pipelines, distributed systems, or jobs that need retry logic — tools like systemd timers, Apache Airflow, or cloud-native schedulers (AWS EventBridge, GCP Cloud Scheduler) are better fits. Knowing when NOT to use cron is as important as knowing how to use it.

# ─────────────────────────────────────────────────────────────────
# CRON NICKNAMES — readable shorthand for common schedules
# ─────────────────────────────────────────────────────────────────

@reboot   /opt/scripts/start_queue_worker.sh   # Run ONCE at system startup
@hourly   /opt/scripts/refresh_cache.sh         # Same as: 0 * * * *
@daily    /opt/scripts/cleanup_sessions.sh      # Same as: 0 0 * * *
@weekly   /opt/scripts/generate_sitemap.sh      # Same as: 0 0 * * 0
@monthly  /opt/scripts/invoice_generator.sh     # Same as: 0 0 1 * *
@yearly   /opt/scripts/archive_old_records.sh   # Same as: 0 0 1 1 *

# ─────────────────────────────────────────────────────────────────
# SYSTEM CRONTAB (/etc/crontab) — has an extra USER field
# ─────────────────────────────────────────────────────────────────
# Format: minute  hour  day  month  weekday  USER  command

# Run log rotation as root every day at 6:25am
25 6 * * *   root    /usr/sbin/logrotate /etc/logrotate.conf

# Run the app's data sync as the 'deploy' user at midnight
0  0 * * *   deploy  /opt/app/bin/sync_data.sh

# ─────────────────────────────────────────────────────────────────
# RUNNING A CRON JOB MANUALLY FOR TESTING
# ─────────────────────────────────────────────────────────────────
# The most reliable way to test a cron job is to simulate
# cron's environment: no shell customizations, minimal PATH.

# Run your script exactly as cron would — bare environment
env -i HOME=/root LOGNAME=root PATH=/usr/bin:/bin \
    /bin/bash /opt/scripts/backup_postgres.sh

# ─────────────────────────────────────────────────────────────────
# CHECKING CRON LOGS — when a job runs but you don't see output
# ─────────────────────────────────────────────────────────────────

# On systemd-based systems (Ubuntu 20+, RHEL 8+)
journalctl -u cron --since "1 hour ago"

# On older systems using syslog
grep CRON /var/log/syslog | tail -50

# On Red Hat / CentOS systems
grep CRON /var/log/cron | tail -50

Debugging Cron Jobs — Diagnosing Silent Failures and Common Pitfalls

When a cron job fails silently, there's no error message, no email, no stack trace — just an application that starts degrading while nobody notices. This is the number one reason cron gets a bad reputation in production. The fix is a repeatable debug workflow.

Start with the cron daemon logs. On modern systems, journalctl -u cron shows every job execution — the exact command, the timestamp, and the user. On older systems, /var/log/syslog or /var/log/cron contains the same. Search for your command name with grep.

If the job ran but had no effect, the problem is usually the environment. Simulate cron's minimal environment with env -i HOME=/root PATH=/usr/bin:/bin /bin/bash your_script.sh. This stripped-down shell will reproduce the exact conditions under which cron runs your script. Any errors you see here are the errors cron sees — they just go unlogged.

Another common pitfall: the script has a shebang pointing to the wrong interpreter. If your script starts with #!/usr/bin/python3 but Python 3 is installed at /usr/local/bin/python3, cron will fail with a misleading error. Always use #!/usr/bin/env python3 for portability.

Finally, remember that cron jobs inherit the umask of their parent process — typically 022. If your job creates files that need specific permissions, set umask explicitly in the script.

# ─────────────────────────────────────────────────────────────────
# DEBUGGING A CRON JOB THAT FAILS SILENTLY
# ─────────────────────────────────────────────────────────────────

# Step 1: Check if cron daemon is even running
ps aux | grep -E 'cron|crond'

# Step 2: List your crontab and verify the schedule
crontab -l

# Step 3: Check cron logs for recent job starts
journalctl -u cron --since '2 hours ago' | grep -E 'CMD|FAILED'

# Step 4: Simulate cron environment and run the script manually
# (Replaces all shell customizations)
env -i HOME="/home/deploy" LOGNAME="deploy" PATH="/usr/bin:/bin" SHELL="/bin/bash" \
    /bin/bash /opt/scripts/your_script.sh

# Step 5: If the script works in Step 4, the issue is PATH or a missing env variable.
# Add explicit exports at the top of the script.

# Step 6: Check file permissions on script and all parent directories
ls -la /opt/scripts/your_script.sh

# Step 7: Test with a very basic cron job to confirm cron itself works
# (Run for 1 minute from now using 'at' or temporary crontab line)
# Then check cron log
echo "* * * * * echo 'cron works' >> /tmp/cron_test.log" | crontab -
# Wait 1 minute then check
cat /tmp/cron_test.log
# Clean up
crontab -r
crontab -l 2>/dev/null || echo "crontab cleaned"

Real-World Cron Patterns — Database Backups, Cache Warming, and Health Checks

In production, cron jobs fall into a handful of common patterns. Getting these right means the difference between a reliable automation pipeline and a pager at 3am.

Database Backups — The most critical cron job. Use the production-ready pattern from section 2: explicit env, logging, locking. Always validate the backup: after pg_dump, run pg_restore --list on the dump file to catch corruption early. Store backups off-server (S3, NFS) and retain multiple copies.

Cache Warming — Many apps rely on a cache that expires at midnight. Instead of serving slow responses to the first users, run a cron job just before peak traffic (e.g., 0 5 *) to pre-compute and populate the cache. Use flock to prevent overlap if the warming takes longer than the interval.

Health Checks — These run every 5 or 15 minutes and report system health to monitoring (e.g., check disk space, process up, API endpoint response). The script should output metrics in a parseable format (JSON) so Prometheus or a custom collector can ingest them. Unlike backups, health checks should NOT use locking — you want to run even if the previous check is still going (but alert if that happens).

Log Rotation — Cron handles log rotation via /etc/cron.daily/logrotate. When building custom rotation for app logs, never use rm -rf — use logrotate with compression and date-stamped filenames. Accidentally removing a log file that a running process is writing to can cause process hang or data loss.

Data Syncing — For ETL or replication between systems, schedule the sync during off-peak hours. Use idempotent scripts (sync only what changed) and monitor for latency. If a sync fails, cron won't retry — wrap it in a retry loop with exponential backoff.

# ─────────────────────────────────────────────────────────────────
# PATTERN 1: Database Backup Validation
# ─────────────────────────────────────────────────────────────────
# Inside the backup script, after dump:
pg_dump -U $DB_USER $DB_NAME | gzip > $BACKUP_PATH.gz

# Validate the backup structure (quick check — doesn't load data)
if ! gzip -dc $BACKUP_PATH.gz | pg_restore --list > /dev/null 2>&1; then
    echo "[$TIMESTAMP] ERROR: Backup validation failed!"
    exit 1
fi

# ─────────────────────────────────────────────────────────────────
# PATTERN 2: Cache Warming (e.g., pre-generate popular product pages)
# ─────────────────────────────────────────────────────────────────
# Scheduled at 5am daily
0 5 * * * /opt/scripts/warm_cache.sh

# warm_cache.sh
#!/bin/bash
for url in $(cat /opt/scripts/top_urls.txt); do
    curl -s -o /dev/null -w "%{http_code}" "$url" >> /var/log/cache_warm.log 2>&1
done

# ─────────────────────────────────────────────────────────────────
# PATTERN 3: Health Check with JSON Output
# ─────────────────────────────────────────────────────────────────
# Every 15 minutes
*/15 * * * * /opt/scripts/health_check.sh

# health_check.sh
#!/bin/bash
cat <<EOF
{
  "timestamp": "$(date -Iseconds)",
  "disk_free_pct": $(df / | awk 'NR==2 {print $5}' | tr -d '%'),
  "mem_free_mb": $(free -m | awk 'NR==2 {print $7}'),
  "nginx_running": $(pgrep nginx | wc -l)
}
EOF
# The output goes to stdout; redirect in crontab to a log or a file readable by monitoring.

What Is a Cron Job? (And Why Your Junior Dev Keeps Calling It 'Cron Job')

A cron job is a single entry in a crontab file that tells the cron daemon to run a command on a schedule. That's it. Nothing magical. The daemon wakes up every minute, scans all the crontabs on the system, matches the current time against every job's schedule, and fires off any that match.

The confusion usually starts when people conflate 'cron' (the daemon) with 'cron job' (one scheduled unit). If you're debugging why a script ran twice, you're not asking about the daemon — you're asking about a specific job definition. The daemon doesn't care about your feelings. It only cares about the crontab format.

Every user has their own crontab file. System-wide jobs live in /etc/crontab and /etc/cron.d/. The distinction matters because environment variables and execution contexts differ. User crontabs run with that user's shell profile. System crontabs require an explicit user field. Get that wrong and your script silently dies because PATH is empty.

// io.thecodeforge — devops tutorial
//
// The actual structure behind the curtains
//
// User crontab (managed by `crontab -e`):
# ┌───────────── minute (0-59)
# │ ┌───────────── hour (0-23)
# │ │ ┌───────────── day of month (1-31)
# │ │ │ ┌───────────── month (1-12)
# │ │ │ │ ┌───────────── day of week (0-6, Sunday=0 or 7)
# │ │ │ │ │
0 3 * * * /usr/local/bin/rotate_logs.sh

// System crontab (/etc/crontab) adds a user column:
# minute hour dom month dow user   command
0       3    *   *     1-5   root   /usr/local/bin/rotate_logs.sh

// /etc/cron.d/ files follow system format:
# Anacron compatible directory for daily tasks
SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
0 3 * * * root /usr/local/bin/rotate_logs.sh 2>/tmp/rotate_err.log

Cron Syntax Explained — The Five Fields That Control Your Autopilot

Every cron job line has exactly five time fields followed by the command. No exceptions. If someone shows you a six-field line, they're either on a system crontab or they've been reading bad blog posts. The fields are: minute, hour, day of month, month, day of week. Each accepts numbers, ranges, lists, and step values.

The asterisk is a wildcard meaning 'every'. So 0 3 means: run at minute 0 of hour 3, every day of every month, regardless of weekday. That's your 3 AM backup slot. Get a comma in there and you have a list: 0 3,15 runs at 3 AM and 3 PM daily.

Ranges use hyphens: 0 9-17 1-5 means every hour on the hour during business hours, Monday through Friday. Steps divide time: /10 fires every ten minutes. Why? Because /10 means 'every 10 minutes starting from 0' — the step operator divides the range and takes every Nth value.

The trap: day-of-week and day-of-month are OR'd together in standard cron, not AND'd. A job scheduled for 0 0 15 * 5 runs on the 15th of every month AND every Friday. If you want strict AND behavior, you need two separate jobs or a wrapper script that checks both conditions.

// io.thecodeforge — devops tutorial
//
// Real schedules you'll actually debug at 3 AM
//
# Every 5 minutes — log rotation watchdog
*/5 * * * * /opt/scripts/check_disk.sh

# Midnight daily — database dump
0 0 * * * /opt/scripts/db_backup.sh --database=production --compress=lz4

# 2:30 PM every Monday — weekly audit report
30 14 * * 1 /opt/scripts/audit_report.sh

# Every 6 hours — sync to cold storage
0 */6 * * * /opt/scripts/sync_to_s3.sh

# Business hours ping — 9:15, 10:15, ..., 17:15 Mon-Fri
15 9-17 * * 1-5 /opt/scripts/health_check.sh

# First day of quarter — archive old logs
0 4 1 */3 * /opt/scripts/archive_quarter.sh