AutoSys Status Codes — PEND_MACH Disk Space Trap

You'll stare at autorep output every day. SU, FA, RU, OH, OI, PE, TE, AC, IN, ST, QU, RE.

That's 12 status codes. Some are normal. Some need immediate action. Some are traps that operations teams misinterpret for hours.

This article covers each status — what it means, what causes it, and exactly what to do. The difference between ON_HOLD and ON_ICE appears in every AutoSys interview. Master it here.

What AutoSys Status Codes Actually Tell You — and What They Hide

AutoSys status codes are integer values (0–255) emitted by jobs at termination, encoding the exit condition as determined by the AutoSys agent. The core mechanic: the agent captures the process exit code and maps it to a status string (SUCCESS, FAILURE, TERMINATED, etc.), but the raw code itself is often lost or overwritten by the agent's own logic. In practice, the most dangerous status is PEND_MACH (pending machine) — it means the job never started because no agent could be assigned. The agent selection algorithm considers machine load, time zone, and operator-defined attributes, but a full disk on the primary agent host silently prevents assignment, leaving the job in PEND_MACH with exit code 0. This is not a job failure; it's an infrastructure failure that looks like a scheduling delay. Senior engineers treat PEND_MACH as a canary for disk space or agent health, not a job issue. Use status codes to distinguish between job logic failures (non-zero exit) and execution failures (PEND_MACH, TERMINATED). In real systems, monitoring PEND_MACH trends catches disk-full scenarios before they cascade.

Normal lifecycle statuses

These are the statuses a job moves through during a healthy execution. You should see them regularly and not be alarmed.

INACTIVE (IN): Job hasn't been triggered yet. Waiting for its schedule or conditions. This is the starting state for most scheduled jobs.

ACTIVATED (AC): The job's parent BOX is in RUNNING state, but the job itself hasn't started yet because its own conditions aren't met. This is normal — it means the box is active and the job is queued.

STARTING (ST): Brief transition state. The Event Processor sent the start command to the agent, and the agent is spinning up the process. Usually lasts under 3 seconds.

RUNNING (RU): Job is executing on the agent machine. Monitor runtime against expected duration. A job that runs 4x longer than usual may be hung.

SUCCESS (SU): Job completed with exit code 0. Normal completion. No action needed.

RESTART (RE): Job is being automatically retried after a failure (n_retrys configured). This is normal for jobs with retry logic. Only be concerned if the final attempt also fails.

# Check status of a single job
autorep -J daily_report

# Check status with full details (start/end times, exit code)
autorep -J daily_report -d

# List all jobs and their current status
autorep -J %

# Filter by status (SU = SUCCESS, FA = FAILURE, RU = RUNNING)
autorep -J % -s SU
autorep -J % -s FA
autorep -J % -s RU

# Check if a job is hung (running longer than expected)
autorep -J long_job -d | grep "Run Time"

All status codes with abbreviations

The autorep command uses two-letter abbreviations for status codes. Here they all are, including the less common ones:

Normal/Interim: AC (ACTIVATED), IN (INACTIVE), ST (STARTING), RU (RUNNING), SU (SUCCESS), RE (RESTART)

Problem/Unusual: FA (FAILURE), PE (PEND_MACH), TE (TERMINATED), QU (QUE_WAIT)

Manual holds: OH (ON_HOLD), OI (ON_ICE)

Less common: DE (DELAYED — job waiting for start time), OP (OPERATOR — operator held), SP (STOPPED), UP (UP STREAM — waiting on upstream), WD (WAIT_DEP — waiting on file watcher)

The difference between ON_HOLD and ON_ICE is the most misunderstood. ON_HOLD + OFF_HOLD = start immediately if conditions met. ON_ICE + OFF_ICE = wait for next scheduling cycle.

AC  ACTIVATED    -- Box is RUNNING but this job hasn't started yet
DE  DELAYED      -- Job waiting to meet its start time criteria
FA  FAILURE      -- Job completed with non-zero exit code
IN  INACTIVE     -- Job hasn't been triggered; waiting for schedule
OH  ON_HOLD      -- Manually held; won't run until released
OI  ON_ICE       -- Suspended; won't run even when conditions reappear
OP  OPERATOR     -- Operator held (sendevent -E HOLDJOB)
PE  PEND_MACH    -- Waiting for target machine (agent offline)
QU  QUE_WAIT     -- Waiting for available machine load/resources
RE  RESTART      -- Job is being restarted after failure
RU  RUNNING      -- Job is currently executing
ST  STARTING     -- Job sent to agent; waiting for execution to begin
SU  SUCCESS      -- Job completed with exit code 0
TE  TERMINATED   -- Job was killed (term_run_time, KILLJOB event, etc.)
UP  UP_STREAM    -- Waiting on upstream dependencies
WD  WAIT_DEP     -- Waiting on file watcher condition

The problem statuses and what to do

These statuses almost always require your attention. Each has a specific diagnosis path.

# ── FAILURE: job failed ──────────────────────────────────────────
# Check the error log first
cat /logs/autosys/daily_report.err
# Then check the event log
autorep -J daily_report -d
# Review recent runs
autorep -J daily_report -run 7
# Retry after fixing the underlying issue
sendevent -E FORCE_STARTJOB -J daily_report

# ── PEND_MACH: agent machine offline ─────────────────────────────
# First: check disk space on agent (MOST COMMON)
ssh prod-server-01 'df -h'
# Then check agent status
autoping -m prod-server-01
autorep -M prod-server-01
# Restart agent after fixing issue
/opt/CA/agent/bin/agent_start

# ── TERMINATED: job was killed ────────────────────────────────────
# Check if killed by term_run_time or manually
autorep -J slow_job -d | grep -E 'term_run_time|KILLJOB'
# If due to term_run_time, increase it:
update_job: slow_job
term_run_time: 180

# ── QUE_WAIT: machine overloaded ──────────────────────────────────
# Check machine load limits
autorep -M machine-name
autorep -J jobname -q | grep load

The Status Transition Matrix: Why Your Job Didn't Fail — It Just Never Started

Most engineers treat AutoSys status codes as final verdicts. They're not. They're snapshots of a sequence that can stall at any point. A job stuck in ACTIVATED for 15 minutes isn't failing — it's waiting on a condition that never evaluated true. A job bouncing between QUEUED and STARTING is a dispatcher bottleneck, not a script error.

The real skill is reading the transition path, not the final status. When you see INACTIVE after a failure, that's the system cleaning house — your job already exhausted its retry limit. When you see TERMINATED with exit code 0, someone sent a kill signal mid-execution. The exit code is meaningless then.

Build a mental model of the state machine. Every status has a permitted predecessor and successor. If you see a transition that shouldn't happen — like SUCCESS flipping to INACTIVE without an intervening failure — you're looking at a force-restart or a database corruption. Both require human intervention, not automation.

// io.thecodeforge — devops tutorial

// Valid transition paths for an AutoSys job
name: payment_reconciliation_job
machine: payment-prod-01
command: /opt/scripts/run_reconciliation.sh
max_run_alarm: 900

# Valid status sequences:
# STARTING -> RUNNING -> SUCCESS
# STARTING -> RUNNING -> FAILURE -> INACTIVE (retries exhausted)
# ACTIVATED -> QUEUED -> STARTING -> RUNNING -> TERMINATED

# The bad one — never seen in logs but happens in prod:
# SUCCESS -> INACTIVE without FAILURE
# This = a manual force-restart or Event Server glitch

dependencies:
  - job: data_ingestion_job
    condition: success

# if data_ingestion_job never starts, payment_reconciliation stays ACTIVATED
# It didn't fail. It never got the signal.

Exit Codes vs. AutoSys Status Codes: The Lie You've Been Told

Here's what nobody tells you: AutoSys status codes and process exit codes are different layers of reality. A job can exit with code 0 (success to the OS) but AutoSys flags it as FAILURE because of exit code mapping. Conversely, a script that dumps core with exit code 139 can be registered as SUCCESS if the job profile says 'ignore_exit_code: 1'.

The exit_code_mapping parameter is a minefield. Default behavior: any non-zero exit is failure. But production systems routinely map exit code 2 to SUCCESS because some old Perl script uses 2 for "no data found" — which is not an error. That's your problem. You're debugging an AutoSys FAILURE that is actually a business logic success.

Stop looking at status codes in isolation. Cross-reference them with the actual exit code from the job's stdout log. If you see FAILURE with exit code 0, you have a timeout issue — the job ran past max_run_alarm. If you see SUCCESS with exit code 137, someone sent SIGKILL to the process and AutoSys swallowed it.

Map your exit codes explicitly. Put it in version control. Treat the defaults as suspicious.

// io.thecodeforge — devops tutorial

// Profile for a job that 'succeeds' when no data exists
job_name: crm_export_loader
machine: crm-worker-03
command: /opt/scripts/load_crm_data.sh

# Default AutoSys behavior: non-zero exit = FAILURE
# This mapping changes the rules:
exit_code_mapping:
  - exit_code: 0
    description: Data loaded successfully
    status: SUCCESS
  - exit_code: 2
    description: No new data available
    status: SUCCESS
  - exit_code: 1
    description: Database connection lost
    status: FAILURE
  - exit_code: 137
    description: Process killed by OOM
    status: TERMINATED

max_run_alarm: 300
# If this runs 301 seconds, AutoSys terminates it
# Exit code from kill signal = 143, not from the script
# Without explicit mapping, you see FAILURE with exit 143