AutoSys Restart Failures — Diagnosis Before Recovery

Job failures in AutoSys are inevitable. What separates good operators from great ones is the speed and thoroughness of their recovery process. This article walks through a complete failure handling workflow — from initial alert to verified recovery — the way experienced AutoSys admins actually do it in production.

Why AutoSys Job Failure Handling Restart Is Not a Retry Button

AutoSys job failure handling restart is a deterministic recovery mechanism that re-runs a failed job from its defined start point, not from where it crashed. The core mechanic: when a job exits with a non-zero status, AutoSys evaluates the job's 'term_run_time' and 'max_run_alarm' attributes, then applies the 'failure_exit' condition to decide whether to trigger a restart. This is not a simple retry — it's a stateful decision that respects job dependencies, box hierarchies, and global conditions.

In practice, the restart behavior is governed by the job's 'max_retry' and 'retry_interval' parameters. If max_retry is set to 3 with a retry_interval of 60 seconds, AutoSys will attempt to restart the job up to 3 times, waiting 60 seconds between each attempt. Critically, the restart does not reset the job's exit code history — the job's 'status' transitions from FAILURE to RESTART, and the 'exit_code' from the failed run persists in the job report. This means downstream jobs that depend on the failed job's exit code may still see the failure unless explicitly handled.

Use this mechanism when a job fails due to transient conditions — network timeouts, resource contention, or temporary file locks — but not for logic errors or data corruption. In real systems, misconfigured restart policies cause cascading failures: a job that fails due to a missing file will keep restarting, consuming resources and delaying manual intervention. The restart is a tactical recovery tool, not a substitute for root cause analysis.

Step 1: Triage — identify the failure

When paged, your first job is to understand the scope before touching anything. Don't jump into the first failed job — check how many others are down. A single failure is a script or dependency issue. A cascade points to an environment problem — machine, database, network, or upstream job chain.

# 1a. How many jobs are failing?
autorep -J % -s FA | wc -l

# 1b. Are jobs failing on a specific machine?
autorep -J % -s FA | awk '{print $1}' | while read j; do
  autorep -J $j -d | grep machine
done

# 1c. Is the agent machine available?
autorep -M %  # check all machines — any MISSING?

# 1d. Are we looking at PEND_MACH instead of FAILURE?
autorep -J % -s PE | wc -l

Step 2: Diagnose — read the error log

Never restart a job without reading the error log first. You'll restart it into the same failure and wonder why. AutoSys captures standard error and standard output. Always check both — sometimes the script succeeds but reports errors to stdout that don't affect exit code.

# 2a. Get the error log path
autorep -J failed_job -d | grep -E 'std_err|std_out'

# 2b. Read the error log
tail -100 /logs/autosys/failed_job.err

# 2c. Read the AutoSys event log for this job
autorep -J failed_job -run 1   # previous run
autorep -J failed_job -d       # detailed current state

# 2d. Check the exit code
autostatus -J failed_job

Step 3: Fix and restart correctly

After fixing the root cause, restart the job using the right command. RESTART is the safe option — it clears the failure status and restarts the job with the same conditions (dependencies, machine requirements). FORCE_STARTJOB bypasses all conditions — use it when you need to start a job that is still blocked by unmet dependencies (e.g., after manually fixing data). For box failures, never restart the box without first resetting the failed inner jobs — otherwise the box will immediately re-fail.

# Option A: RESTART (for failed jobs — clean retry)
sendevent -E RESTART -J failed_job

# Option B: FORCE_STARTJOB (bypasses conditions — use when needed)
sendevent -E FORCE_STARTJOB -J failed_job

# Option C: If a BOX failed — restart the box and all its inner jobs
# First, reset all inner FAILURE jobs:
sendevent -E CHANGE_STATUS -J inner_job1 -s INACTIVE
sendevent -E CHANGE_STATUS -J inner_job2 -s INACTIVE
# Then restart the box:
sendevent -E FORCE_STARTJOB -J eod_box

# Monitor the restart
watch -n 10 'autorep -J failed_job'

Step 4: Verify recovery — beyond the SUCCESS status

A job showing SUCCESS in AutoSys means its exit code was 0. It does not mean the data is correct or the downstream chain completed. Verification is the step that separates reliable operators from ones who get called back an hour later. Check that all downstream jobs in the chain succeeded, and then validate the actual output: file size, row count, or a simple data check.

# Check all downstream jobs in the chain
autorep -J eod_%

# Confirm the box moved to SUCCESS
autorep -J eod_processing_box

# Check the actual output — did the script produce the expected file?
ls -la /data/output/daily_report_$(date +%Y%m%d).csv
wc -l /data/output/daily_report_$(date +%Y%m%d).csv  # check row count

Step 5: Post-recovery actions — document and escalate

Recovery isn't complete until you've recorded what happened and why. This step prevents recurring failures and helps your team learn. Write a brief incident summary: what failed, root cause, fix applied, time to recover, and any follow-up needed (e.g., increase term_run_time, add a healthcheck to the database monitoring). If the failure indicates a systemic problem (e.g., multiple jobs using the same broken dependency), escalate to the appropriate team. Finally, update any runbooks or monitoring rules that could have caught the issue earlier.

# Log the incident to a shared file or alerting system
echo "$(date): Job eod_extract failed — DB_PROD unreachable" >> /var/log/autosys_incidents.log
echo "Root cause: Network switch failure. Fixed by ops team." >> /var/log/autosys_incidents.log
echo "Recovery time: 12 min" >> /var/log/autosys_incidents.log

# If needed, update the job's term_run_time or add a retry policy
sendevent -E CHANGE_STATUS -J eod_extract -a term_run_time=1200

# Notify stakeholder
mail -s "AutoSys incident report: eod_extract" ops-team@company.com < /var/log/autosys_incidents.log

The 3AM Blind Spot: Why a SUCCESS Exit Code Can Lie to You

You fixed the job. Restarted it. Exit code 0. Good night, right? Wrong. A SUCCESS status only tells you the shell command didn't crash. It does not tell you the data landed in the right table, or that the file transfer completed without corruption, or that your downstream job didn't consume garbage.

In production, I've seen AutoSys jobs exit cleanly while writing to a full disk, hitting a stale symlink, or processing yesterday's snapshot instead of today's. The OS says it's fine. AutoSys says it's fine. But your business logic just silently robbed a bank.

The fix: always add a post-success validation command inside the job definition. Use validate_cmd or chain a lightweight verification script that checks checksums, row counts, or API responses. Never trust the exit code alone. Trust your validation layer.

// io.thecodeforge — devops tutorial
// This job validates data integrity after the main process

job_name: data_pipeline_validate
  machine: prod_app_server_01
  command: /opt/scripts/validate_incremental_load.sh
  
  // Run only after main job finishes, regardless of exit?
  // No. Only run if main job exits 0.
  condition: s(data_pipeline_main)
  
  // If validation fails, fail this job loudly
  failure_exit_codes: 10-99
  
  // Capture both stdout and stderr for forensic log
  std_out_file: /var/log/autosys/validation_<JIID>.out
  std_err_file: /var/log/autosys/validation_<JIID>.err
  
  // Notify on validation failure, not on the main job
  notification: mailx -s "FAILED: data_pipeline_validate" ops@company.com

The AutoSys Auto-Restart Trap: When "Retry" Is Sabotage

Every junior's first instinct: enable auto-retry in the job definition so the system handles it. Stop. Auto-retry is not a recovery strategy. It is a bandage that masks transient failures and turns permanent ones into infinite loops.

Here's the rule: auto-retry only for infrastructure hiccups — network timeouts, disk pressure, DNS failures. Never for logic errors, data corruption, or missing dependencies. If your job fails because the input file is malformed, retrying it 47 times will never fix the file. It just screams louder.

Config your auto-retry with max_retry: 3 and a backoff formula that doubles the interval. Monitor the retry count as an alert. If a job hits retry #3, don't restart — escalate. Write a wrapper that exits with a non-standard code (e.g., 127) for logic failures, then set failure_exit_codes: 127 to disable auto-retry for those cases.

// io.thecodeforge — devops tutorial
// Intelligent retry with failure code discrimination

job_name: ingestion_api_call
  machine: prod_batch_01
  command: /opt/scripts/ingest_partner_data.sh
  
  // Only retry on specific exit codes (infra failures)
  failure_exit_codes: 0-10, 70-99
  max_retry: 3
  retry_intervals: 30, 60, 120   // exponential backoff: 30s, 60s, 120s
  
  // Logic failures (code 127) must NOT be retried
  // They go straight to manual escalation
  
  std_out_file: /var/log/autosys/ingest_<JIID>.out
  std_err_file: /var/log/autosys/ingest_<JIID>.err
  
  // Only send notification after final retry failure
  notification: echo "Job ${AUTO_JOB_NAME} failed after 3 retries" | mailx -s "AUTO-RETRY EXHAUSTED" oncall@team.com