AutoSys Job Stuck in STARTING: Why It Never Runs and How to Release It

Forty jobs on one machine, all in STARTING, none running. Force-starting them just puts them back in STARTING. That's because STARTING isn't about the job at all: it's the scheduler waiting for the agent to answer. This guide shows what the state means, how to find why the agent isn't answering, and how to release the jobs.

What STARTING Means

When a job's conditions are met, the scheduler sends a start request to the agent on the job's machine and sets the job to STARTING. When the agent has launched the process it reports back, and the job moves to RUNNING. A job that stays in STARTING is one where that acknowledgement never arrived. The script hasn't run, so its log won't help you.

# Jobs currently in STARTING
autorep -J ALL | grep ' ST '

# Which machine are they on?
autorep -J FIN_LOAD_01 -q | grep machine

Cause 1: The Agent Is Down or Unreachable

If the agent service is stopped, the host is down, or a firewall blocks the agent's port, the start request never gets a reply. autoping tests the round trip from the scheduler to the agent and back, so it's the first command to run.

autoping -m finbatch01

Cause 2: The Agent's Disk Is Full

The agent writes spool files, job logs and its own logs before it can launch a job. When that filesystem is 100% full, it can't, so it never acknowledges the start. This is the cause behind many 'everything on one box is stuck' incidents, and it's often the agent's own logs that filled the disk.

# On the agent machine
df -h /opt
du -sh /opt/CA/WorkloadAutomationAE/SystemAgent/*/log | sort -h | tail -3

Cause 3: The Agent Is Overloaded

Machines can be given a limit on how many jobs they run at once. When the agent is at that limit, or the host is overloaded, new jobs wait. The signature is different from the other causes: some jobs on the machine start, and others wait in line.

# Machine definition and its limits
autorep -M finbatch01 -q

# Jobs running on it right now
autorep -J ALL | grep ' RU ' | wc -l

Release the Stuck Jobs

Freeing space or restoring the network doesn't move jobs that are already in STARTING: the scheduler is still waiting for an answer that won't come. Restart the agent so it starts clean, then reset each stuck job's status and start it again. Check first that the job didn't actually run, so you don't run a load twice.

# After fixing the cause and restarting the agent
for j in $(autorep -J FIN_LOAD% | awk '$NF=="ST"{print $1}'); do
  sendevent -E CHANGE_STATUS -s FAILURE -J "$j"
  sendevent -E FORCE_STARTJOB -J "$j"
done