SUCCESS (SU): Job completed with exit code 0. Normal. No action.
FAILURE (FA): Job exited non-zero. Check std_err_file before restarting.
RUNNING (RU): Job executing. Monitor runtime vs expected duration.
PEND_MACH (PE): Agent unreachable. First check: disk space on agent (df -h).
ON_HOLD (OH): Manually held. OFF_HOLD starts immediately if conditions met.
ON_ICE (OI): Suspended. OFF_ICE waits for next scheduling cycle.
ACTIVATED (AC): Job inside RUNNING box, waiting its turn. Normal.
TERMINATED (TE): Job killed (term_run_time or KILLJOB). Investigate why.
Production rule: 20+ jobs in PEND_MACH on same machine = disk full. Clear space, restart agent.
Plain-English First
AutoSys job statuses are like package tracking codes — each status tells you exactly where your job is in its lifecycle. RUNNING means it's on the truck. SUCCESS means it was delivered. FAILURE means something went wrong. PEND_MACH means the truck broke down.
You'll stare at autorep output every day. SU, FA, RU, OH, OI, PE, TE, AC, IN, ST, QU, RE.
That's 12 status codes. Some are normal. Some need immediate action. Some are traps that operations teams misinterpret for hours.
This article covers each status — what it means, what causes it, and exactly what to do. The difference between ON_HOLD and ON_ICE appears in every AutoSys interview. Master it here.
Normal lifecycle statuses
These are the statuses a job moves through during a healthy execution. You should see them regularly and not be alarmed.
INACTIVE (IN): Job hasn't been triggered yet. Waiting for its schedule or conditions. This is the starting state for most scheduled jobs.
ACTIVATED (AC): The job's parent BOX is in RUNNING state, but the job itself hasn't started yet because its own conditions aren't met. This is normal — it means the box is active and the job is queued.
STARTING (ST): Brief transition state. The Event Processor sent the start command to the agent, and the agent is spinning up the process. Usually lasts under 3 seconds.
RUNNING (RU): Job is executing on the agent machine. Monitor runtime against expected duration. A job that runs 4x longer than usual may be hung.
SUCCESS (SU): Job completed with exit code 0. Normal completion. No action needed.
RESTART (RE): Job is being automatically retried after a failure (n_retrys configured). This is normal for jobs with retry logic. Only be concerned if the final attempt also fails.
check_status.shBASH
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# Check status of a single job
autorep -J daily_report
# Check status with full details (start/end times, exit code)
autorep -J daily_report -d
# List all jobs and their current status
autorep -J %
# Filter by status (SU = SUCCESS, FA = FAILURE, RU = RUNNING)
autorep -J % -s SU
autorep -J % -s FA
autorep -J % -s RU
# Checkif a job is hung (running longer than expected)
autorep -J long_job -d | grep "Run Time"
The Event Processor is the dispatch center. Agents are the delivery trucks.
Production Insight
A team saw a job in ACTIVATED status for 3 hours and assumed it was stuck. They force-started it. The job ran and failed because its dependency hadn't completed yet. The original job would have started 10 minutes later when the dependency finished.
ACTIVATED is not stuck. It means 'waiting for conditions inside a running box'. The job was fine. The team's impatience caused a failure.
Rule: ACTIVATED only becomes a problem if the parent box has been RUNNING for longer than the expected total duration of all jobs in the box. Otherwise, it's normal.
Key Takeaway
IN → AC → ST → RU → SU is normal job lifecycle.
ACTIVATED means 'waiting inside running box', not stuck.
IfJob in ACTIVATED for < 1 hour, box still RUNNING
→
UseNormal. Job is waiting for its conditions inside the box.
IfJob in RUNNING for > 2x expected duration
→
UseProbably hung. Investigate. May need KILLJOB.
IfJob in PEND_MACH, other jobs on same machine also PEND_MACH
→
UseProblem. Agent machine likely down or disk full. Check immediately.
IfJob in FAILURE, first time in weeks
→
UseProblem. Investigate root cause before restarting.
thecodeforge.io
AutoSys Job Status Lifecycle — All Status Codes
Autosys Job Status Codes
All status codes with abbreviations
The autorep command uses two-letter abbreviations for status codes. Here they all are, including the less common ones:
Normal/Interim: AC (ACTIVATED), IN (INACTIVE), ST (STARTING), RU (RUNNING), SU (SUCCESS), RE (RESTART)
Problem/Unusual: FA (FAILURE), PE (PEND_MACH), TE (TERMINATED), QU (QUE_WAIT)
Manual holds: OH (ON_HOLD), OI (ON_ICE)
Less common: DE (DELAYED — job waiting for start time), OP (OPERATOR — operator held), SP (STOPPED), UP (UP STREAM — waiting on upstream), WD (WAIT_DEP — waiting on file watcher)
The difference between ON_HOLD and ON_ICE is the most misunderstood. ON_HOLD + OFF_HOLD = start immediately if conditions met. ON_ICE + OFF_ICE = wait for next scheduling cycle.
status_abbreviations.txtBASH
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
ACACTIVATED -- Box is RUNNING but this job hasn't started yet
DEDELAYED -- Job waiting to meet its start time criteria
FAFAILURE -- Job completed with non-zero exit code
ININACTIVE -- Job hasn't been triggered; waiting for schedule
OH ON_HOLD -- Manually held; won't run until released
OI ON_ICE -- Suspended; won't run even when conditions reappear
OPOPERATOR -- Operatorheld (sendevent -E HOLDJOB)
PE PEND_MACH -- Waitingfor target machine (agent offline)
QU QUE_WAIT -- Waitingfor available machine load/resources
RERESTART -- Job is being restarted after failure
RURUNNING -- Job is currently executing
STSTARTING -- Job sent to agent; waiting for execution to begin
SUSUCCESS -- Job completed with exit code 0TETERMINATED -- Job was killed (term_run_time, KILLJOB event, etc.)
UP UP_STREAM -- Waiting on upstream dependencies
WD WAIT_DEP -- Waiting on file watcher condition
ON_HOLD vs ON_ICE — the most misunderstood difference
ON_HOLD: OFF_HOLD starts job immediately if conditions are currently satisfied. ON_ICE: OFF_ICE does NOT start immediately — it waits for conditions to reoccur in the next scheduling cycle. This difference causes major operational confusion.
Production Insight
A financial firm put a midnight reporting job ON_ICE at 2 PM to prevent it from running during database maintenance. After maintenance finished at 4 PM, they did OFF_ICE expecting it to run immediately. It didn't. The job ran at midnight as originally scheduled. The report was 8 hours late.
The team confused ON_ICE with ON_HOLD. They should have used ON_HOLD for a temporary pause during business hours.
Rule: Use ON_HOLD when you want the job to run as soon as you're ready. Use ON_ICE when you want the job to return to its normal schedule without running out-of-cycle.
Use ON_HOLD for temporary pauses (maintenance, migrations).
Use ON_ICE for permanent schedule changes or to prevent out-of-cycle runs.
The problem statuses and what to do
These statuses almost always require your attention. Each has a specific diagnosis path.
problem_statuses.shBASH
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
# ── FAILURE: job failed ──────────────────────────────────────────
# Check the error log first
cat /logs/autosys/daily_report.err
# Then check the event log
autorep -J daily_report -d
# Review recent runs
autorep -J daily_report -run 7
# Retry after fixing the underlying issue
sendevent -E FORCE_STARTJOB -J daily_report
# ── PEND_MACH: agent machine offline ─────────────────────────────
# First: check disk space on agent (MOSTCOMMON)
ssh prod-server-01'df -h'
# Then check agent status
autoping -m prod-server-01
autorep -M prod-server-01
# Restart agent after fixing issue
/opt/CA/agent/bin/agent_start
# ── TERMINATED: job was killed ────────────────────────────────────
# Checkif killed by term_run_time or manually
autorep -J slow_job -d | grep -E 'term_run_time|KILLJOB'
# If due to term_run_time, increase it:
update_job: slow_job
term_run_time: 180
# ── QUE_WAIT: machine overloaded ──────────────────────────────────
# Check machine load limits
autorep -M machine-name
autorep -J jobname -q | grep load
Output
FAILURE — Error in /logs/autosys/daily_report.err:
ERROR: Database connection timeout after 30s
PEND_MACH — prod-server-01: MISSING (agent service down)
Disk usage on prod-server-01: /apps 100% full — agent stopped
TERMINATED — term_run_time: 60 (exceeded). Job ran for 62 minutes.
QUE_WAIT vs PEND_MACH — subtle but important difference
PEND_MACH = agent unreachable (machine down or agent stopped). QUE_WAIT = agent reachable but machine too busy (load exceeds max_load). QUE_WAIT resolves automatically when load drops. PEND_MACH requires intervention.
Production Insight
A team saw a job in QUE_WAIT and assumed the agent was down. They restarted the agent, which cleared the queue, but the underlying high load was caused by a different job consuming all CPU. The high-load job continued to run, and QUE_WAIT returned immediately.
QUE_WAIT doesn't mean 'agent broken'. It means 'machine is too busy right now'. AutoSys is protecting the machine from overload.
Diagnosis: autorep -M machine-name shows the max_load setting and current load. If load > max_load, jobs wait.
Fix: Increase max_load on the machine definition, or reduce load from other jobs. Don't restart the agent — that doesn't solve the load issue.
FAILURE: Check error log first. Never restart without investigating.
PEND_MACH: First check disk space (df -h). 90% of cases.
TERMINATED: term_run_time or KILLJOB. Increase runtime or fix hang.
QUE_WAIT: Machine overload. Increase max_load or reduce load.
● Production incidentPOST-MORTEMseverity: high
The 2 AM PEND_MACH That Wasn't a Network Problem
Symptom
autorep shows 47 jobs all with PE status on the same agent machine. autoping -m agent-host returns MISSING. No recent changes to network or firewall.
Assumption
The engineer assumed network or firewall because 'agent unreachable' sounds like connectivity. They didn't check the most common cause first.
Root cause
The Remote Agent service writes logs to /var/log/autosys. When /var fills 100%, the agent process stops. The agent doesn't crash with an error — it just exits. AutoSys's Event Processor keeps trying to contact it, sees no response, and leaves jobs in PEND_MACH.
The agent machine had a 20GB /var partition. Log rotation failed 3 days ago. The partition filled completely.
Fix
1. SSH to agent machine: ssh agent-host
2. df -h → /var 100% used
3. Find large files: du -sh /var/log/* | sort -h
4. Rotate or delete old logs: sudo logrotate -f /etc/logrotate.conf
5. Restart agent: /opt/CA/WorkloadAutomationAE/agent/bin/agent_start
6. Verify: autoping -m agent-host → ACTIVE
7. Jobs will auto-resume from PEND_MACH to RUNNING
Prevention: Set up disk space monitoring at 80% for agent machines. Auto-rotate logs weekly.
Key lesson
PEND_MACH on multiple jobs → same machine → disk space is #1 cause.
Diagnosis order: disk space → agent process → network → everything else.
df -h is the first command, not the last. It saves hours.
Agent machines need disk monitoring. Full disk stops the agent silently.
Production debug guideFA, PE, TE, OH/OI — the statuses that need your attention4 entries
Symptom · 01
FAILURE (FA) — job exited with non-zero code
→
Fix
1. Check std_err_file in job definition. 2. autorep -J JOB -L 10 (last runs). 3. Fix root cause (code, missing file, permission). 4. FORCE_STARTJOB after fix.
A job is put ON_ICE during maintenance. After maintenance, OFF_ICE is issued. The job doesn't start. Team assumes AutoSys is broken. The job runs at its scheduled time hours later.
Fix
ON_HOLD: release starts immediately if conditions met. ON_ICE: release waits for next scheduling cycle. Use ON_HOLD for temporary pauses during business hours.
×
Not checking PEND_MACH machine disk space first
Symptom
Engineer spends 2 hours checking network routes, firewalls, agent config. The cause is a 100% full /var partition. df -h would have shown it in 10 seconds.
Fix
For PEND_MACH, always run df -h on the agent machine first. Full disk is the cause in 90% of cases. Only then check network and agent process.
×
Restarting jobs in FAILURE without first reading the error log
Symptom
Job fails. Engineer force-starts it immediately. It fails again. Repeat 5 times. The error log shows 'database connection timeout'. The database is down. Restarting won't help.
Fix
Before any restart, check std_err_file or autorep -L. Understand why it failed. Fix the root cause, then restart once.
×
Not distinguishing INACTIVE from ON_ICE
Symptom
Team sees a job in INACTIVE and thinks it's frozen. They force-start it, but it was just waiting for its schedule. It runs early and processes incomplete data.
Fix
INACTIVE is normal — the job hasn't met its start conditions yet. It's not an error. Check start_times and conditions before intervening.
×
Treating ACTIVATED as a stuck state
Symptom
Job in ACTIVATED for 2 hours. Team force-starts it. The job's dependency wasn't ready. The job fails. The original job would have started correctly 10 minutes later.
Fix
ACTIVATED means 'waiting inside a running box'. It's not stuck unless the parent box has been running longer than the total expected duration of all jobs in the box.
INTERVIEW PREP · PRACTICE MODE
Interview Questions on This Topic
Q01SENIOR
What does PEND_MACH status mean in AutoSys?
Q02SENIOR
What is the difference between ON_HOLD and ON_ICE in AutoSys?
Q03JUNIOR
What does ACTIVATED status mean for a job inside a BOX?
Q04SENIOR
How do you force-start a job that is in FAILURE status?
Q05SENIOR
What is the most common cause of jobs going to PEND_MACH in bulk?
Q06SENIOR
What does QUE_WAIT status mean and how is it different from PEND_MACH?
Q01 of 06SENIOR
What does PEND_MACH status mean in AutoSys?
ANSWER
PEND_MACH (PE) means the job is waiting for its target machine to become available. The Remote Agent on that machine is not responding to the Event Processor.
Most common cause (90% of cases): The agent machine's filesystem is 100% full, causing the agent service to stop.
Other causes:
- Agent service not running
- Machine offline
- Network issue (firewall blocking port 7520)
- Agent machine overloaded (different from QUE_WAIT)
Diagnosis order:
1. SSH to agent machine: df -h (check disk space)
2. Check agent process: ps -ef | grep autosys
3. Test connectivity: autoping -m agent-hostFix: Clear disk space, restart agent, or fix network issue.
Q02 of 06SENIOR
What is the difference between ON_HOLD and ON_ICE in AutoSys?
ANSWER
This is the most common AutoSys interview question.
ON_HOLD (OH): Manual hold. When released (OFF_HOLD), the job starts IMMEDIATELY if its starting conditions are currently satisfied.
ON_ICE (OI): Manual freeze. When released (OFF_ICE), the job does NOT start immediately. It waits for conditions to be re-evaluated in the next scheduling cycle.
Example: A job that runs at midnight is put ON_ICE at 2 PM. At 3 PM, it's released. It runs at midnight, not at 3 PM.
When to use:
- ON_HOLD: Temporary pauses during business hours (database migration, system maintenance) where you want immediate resume
- ON_ICE: Permanent schedule changes or when you don't want out-of-cycle runs
Key difference: OFF_HOLD triggers immediate evaluation. OFF_ICE does not.
Q03 of 06JUNIOR
What does ACTIVATED status mean for a job inside a BOX?
ANSWER
ACTIVATED (AC) means the job's parent BOX is in RUNNING state, but the job itself hasn't started yet because its own individual conditions (start_times, condition dependencies) haven't been met.
ACTIVATED is completely normal. It is NOT an error or a stuck state.
Example: A box starts at 6 PM. Inside the box, Job A has condition: success(extract_job). If extract_job hasn't finished yet, Job A shows ACTIVATED. When extract_job finishes, Job A will automatically transition to STARTING and then RUNNING.
When to investigate: Only if the parent box has been RUNNING for longer than the total expected duration of all jobs in the box. Otherwise, be patient.
Q04 of 06SENIOR
How do you force-start a job that is in FAILURE status?
ANSWER
Step-by-step process:
Step 1 — Investigate (do NOT force-start immediately):
- autorep -J jobname -q | grep std_err_file — locate error log
- cat /path/to/error/file — understand the failure
- autorep -J jobname -L 10 — see recent runs
Step 2 — Fix root cause:
- Code bug → fix and redeploy
- Missing file → request from upstream
- Permission error → engage sysadmin
- Transient network issue → may not need fix
Step 3 — Restart:
- sendevent -E FORCE_STARTJOB -J jobname
- This bypasses all conditions (time and dependencies)
- Job runs immediately
Alternative for retries: If the failure was transient and you just need to retry, sendevent -E RESTART -J jobname is cleaner (signals intent as a retry).
Never: Force-start a failed job without understanding why it failed. You'll likely just see it fail again.
Q05 of 06SENIOR
What is the most common cause of jobs going to PEND_MACH in bulk?
ANSWER
A full disk on the agent machine.
When 20+ jobs on the same machine all show PEND_MACH simultaneously, the agent machine's filesystem is almost certainly 100% full. The Remote Agent service stops when it can't write logs, and AutoSys marks all jobs targeting that machine as PEND_MACH.
Diagnosis:
``bash
ssh agent-host 'df -h'
`
Look for any partition at 100% usage, especially /var, /tmp, or the agent install directory.
Fix:
1. Clear space (rotate logs, delete temp files)
2. Restart agent: /opt/CA/agent/bin/agent_start`
3. Jobs will automatically resume from PEND_MACH
Prevention: Set up disk space monitoring at 80% for all agent machines. Configure automatic log rotation.
Other less common causes: agent process crashed, machine offline, network partition. But always check disk space first — it's the cause 90% of the time.
Q06 of 06SENIOR
What does QUE_WAIT status mean and how is it different from PEND_MACH?
ANSWER
QUE_WAIT (QU) means the target machine is reachable but currently too busy to accept new jobs. AutoSys is protecting the machine from overload.
Difference from PEND_MACH:
- PEND_MACH: Agent unreachable (machine down, agent stopped, network issue). Requires intervention.
- QUE_WAIT: Agent reachable, but machine load > max_load. Resolves automatically when load drops.
Cause: The machine's max_load attribute (default 1.0) is exceeded by the current system load average.
Check:
``bash
autorep -M machine-name # shows max_load and current load
`Resolution:
- Do nothing — jobs will start when load drops below max_load
- Or increase max_load: update_machine: machine-name, max_load: 2.0`
- Or reduce load from other jobs
Common mistake: Restarting the agent when seeing QUE_WAIT. This doesn't help — the load is the issue, not the agent.
01
What does PEND_MACH status mean in AutoSys?
SENIOR
02
What is the difference between ON_HOLD and ON_ICE in AutoSys?
SENIOR
03
What does ACTIVATED status mean for a job inside a BOX?
JUNIOR
04
How do you force-start a job that is in FAILURE status?
SENIOR
05
What is the most common cause of jobs going to PEND_MACH in bulk?
SENIOR
06
What does QUE_WAIT status mean and how is it different from PEND_MACH?
SENIOR
FAQ · 6 QUESTIONS
Frequently Asked Questions
01
What does PEND_MACH mean in AutoSys?
PEND_MACH (PE) means the job is waiting for its target machine to become available. The Remote Agent on the target machine is not responding — usually because the agent service has stopped (commonly due to a full disk), the machine is offline, or there's a network issue.
First diagnostic step: SSH to the agent machine and run df -h. Full disk is the cause in 90% of cases. Clear space and restart the agent.
Was this helpful?
02
What does ACTIVATED mean in AutoSys?
ACTIVATED (AC) means the job's parent BOX is in RUNNING state, but the job itself hasn't started yet because its own conditions (start_times, condition attribute) haven't been met. It's a normal state — the job is queued inside a running box. Do NOT force-start a job in ACTIVATED unless the parent box has been running far longer than expected.
Was this helpful?
03
What is the difference between ON_HOLD and ON_ICE?
ON_HOLD: When you release a job from ON_HOLD (OFF_HOLD), it will run if its starting conditions are currently satisfied. ON_ICE: When you take a job off ICE (OFF_ICE), it will NOT run immediately — it waits until its conditions reoccur in the next scheduling cycle. ON_ICE is a stronger suspension.
Example: A midnight job put ON_ICE at 2 PM and released at 3 PM runs at midnight, not 3 PM. The same job put ON_HOLD at 2 PM and released at 3 PM would run immediately at 3 PM (if conditions are met).
Do NOT restart without investigating — you'll likely just see it fail again immediately.
Was this helpful?
05
What does TERMINATED status mean in AutoSys?
TERMINATED (TE) means the job was killed. Common causes
term_run_time exceeded (AutoSys killed it after max runtime)
Manual KILLJOB event sent by operator
Agent machine went down while job was running
Check autorep -J jobname -d | grep -E 'term_run_time|KILLJOB' to see the cause. If term_run_time was too low, increase it. If manual kill, understand why the job needed killing. After fixing, use RESTART or FORCE_STARTJOB to rerun.
Was this helpful?
06
What does QUE_WAIT mean and how is it different from PEND_MACH?
QUE_WAIT (QU) means the machine is reachable but currently too busy to accept new jobs — the system load exceeds the machine's max_load setting. AutoSys queues the job until load drops.