AutoSys Architecture — Full /var Crashes Agents Silently

Before you write a single line of JIL or schedule your first job, it helps to understand how AutoSys actually works under the hood. The architecture is straightforward but knowing what each component does — and why — will save you a lot of head-scratching when things go wrong in production.

AutoSys has four major components that work together: the Event Server, the Event Processor, Remote Agents, and client tools. Each has a clear job, and understanding the flow between them makes debugging much easier.

By the end you'll know exactly how job definitions flow from JIL to Event Server to Event Processor to Remote Agent and back. You'll understand what PEND_MACH means and why it's the most common production issue. And you'll know the component that, when it fails, stops all job scheduling.

Why AutoSys Agents Fail Silently on Full /var

AutoSys architecture is a distributed job scheduling system where a central Event Processor (the 'scheduler') communicates with Remote Agents running on target machines. The core mechanic: agents poll the Event Processor for work, execute commands, and report status via log files written to /var/log/autosys. When /var fills up, agents cannot write logs or status updates. They do not crash with an error — they simply stop reporting, appearing as 'OFFLINE' or 'UNREACHABLE' in the GUI, while the scheduler assumes they are still alive and continues dispatching jobs. This silent failure is the most common cause of 'lost' jobs in production. In practice, agents use a heartbeat mechanism (default 60-second interval) to signal liveness. A full /var prevents heartbeat log writes, so the Event Processor marks the agent as down after missing 3 consecutive heartbeats. However, the agent process itself remains running — it just cannot communicate. This creates a zombie state: the agent appears active on the host (ps shows it), but the scheduler sees it as dead. Monitoring /var usage is not optional; a threshold of 85% should trigger alerts. Use this architecture when you need centralized control over thousands of jobs across heterogeneous servers. The silent failure mode matters because it breaks the fundamental contract of distributed scheduling: reliable status reporting. Without disk space monitoring, teams waste hours debugging phantom network issues.

The Event Server — the source of truth

The Event Server is a relational database (typically Sybase or Oracle) that stores everything AutoSys needs to operate. This includes all job definitions (what to run, when, where, under which conditions), all events that have occurred (job started, job succeeded, job failed), global variable values, machine definitions, calendar definitions, and monitor and report definitions.

When a job finishes and reports its status, that status goes into the Event Server. When the Event Processor needs to know whether a dependent job's condition is met, it queries the Event Server. It's the single source of truth for the entire AutoSys environment.

The Event Processor — the brain

The Event Processor (also called the scheduler or the event daemon) is the most important component. It runs continuously, polling the Event Server for events. When it detects that a job's starting conditions are met — the right time has arrived, dependent jobs have succeeded, the machine is available — it triggers the job to run on the appropriate agent.

The Event Processor also handles time-based scheduling, evaluates job condition logic, and manages the overall state machine for each job. On Unix/Linux it's started with the eventor command. There is only ever one Event Processor running per AutoSys instance.

# Start the AutoSys event processor (UNIX only)
eventor

# Check if AutoSys components are up
autoping

# Check AutoSys flags and system status
autoflags -a

Remote Agents — where the work actually happens

Remote Agents run on every machine where AutoSys needs to execute jobs. When the Event Processor decides a job should run, it sends a message to the Remote Agent on the target machine. The agent starts the process, monitors it, captures the exit code, and reports the result back to the Event Server.

Agents can be extended with plugins for specific integrations — SAP, Oracle E-Business, PeopleSoft, and others. If an agent goes down, jobs that are supposed to run on that machine go into PEND_MACH status and wait until the agent comes back up.

Client tools — how you interact with AutoSys

Client tools are the interfaces you use to define, manage, and monitor jobs. The main ones are: jil — the command-line JIL processor for creating and modifying job definitions; autorep — reports job status and definitions; sendevent — manually triggers events like starting a job or putting it on hold; autostatus — checks the current status of a specific job; and the WCC Web UI — a browser-based dashboard for monitoring job flows visually.

Most experienced AutoSys administrators work primarily from the command line using jil, autorep, and sendevent. The GUI is useful for monitoring and for people less comfortable with CLI.

# Check status of a specific job
autostatus -J daily_report

# Get a detailed report on a job
autorep -J daily_report -d

# List all jobs in a box
autorep -J box_name%

# Check machine status
autorep -M prod-server-01

The Scheduler — Where Time Becomes a Trigger

You think a cron job is reliable? AutoSys Scheduler doesn't think so. It's the component that turns a wall-clock moment into a job start condition. But here's the rub: it's not a clock, it's a state machine.

The Scheduler runs inside the Event Processor. It doesn't fire jobs. It creates STARTING events. Those events get queued into the Event Server. If the Event Server can't accept the write — full disk, slow I/O, network partition — that STARTING event vaporizes. No retry. No log. Your job simply never runs.

Second trap: the Scheduler uses the system timezone of the machine hosting the Event Processor. If you migrate the Event Processor to a new host and forget to sync timezone, every scheduled job shifts by hours. The Event Server stores the UTC timestamp, but the comparison logic runs in local time. This is not daylight saving ignorance. This is production downtime.

Why this matters: when you see a job that should have fired at 02:00 but didn't, don't immediately blame the Agent. Check Scheduler health, check timezone, check if the Event Server was accepting writes at that second. Time-based triggers are only as reliable as the full write-path to the Event Server.

// io.thecodeforge — devops tutorial

// Check what time AutoSys thinks it is
USER> autostatus -A | grep -i scheduler
SCHEDULER: Running on host 'prod-ep-01', PID 7723
   Current Local Time: 2025-03-18 14:22:35 EDT
   Event Processor TZ: America/New_York
   Event Server UTC:   2025-03-18 18:22:35

// If local and UTC drift by more than 1 second, you have a problem
// Compare with: date +%s on the Event Server host
USER> date +%s; echo "---"; sqlplus -S autosys/autosys@EVENTDB <<< "SELECT TO_CHAR(SYSTIMESTAMP, 'YYYY-MM-DD HH24:MI:SS TZR') FROM DUAL;"
1742322155
---
2025-03-18 14:22:35 EDT

// Off by 0 seconds — good. Off by 3600? You just lost an hour of schedules.

The Dependency Graph Without the Graph — Why Job A Doesn't Care About Job B

You set 'condition: p(jobB)' on job A. You think: when job B finishes, job A starts. Wrong. AutoSys doesn't maintain a real-time dependency tree. It evaluates conditions at job submission time, not continuously.

Here's how it actually works: job A is submitted to the Event Server with a condition. The Event Processor sees the condition, looks up the current status of job B from the Event Server. If job B is SUCCESS, the Event Processor creates a STARTING event for job A immediately. If job B is still RUNNING, the Event Processor does nothing. It does NOT poll. It does NOT watch. The condition sits dead until job B finishes and its status changes.

When job B finishes, job A doesn't automatically start. That status change triggers the Event Processor to re-evaluate ALL conditions that reference job B. This is the "status-change cascade." Every finished job forces a condition re-evaluation across every dependency. If you have 10,000 jobs depending on one master job, that master job finishing will spawn 10,000 condition checks in a single CPU-bound loop. On a busy Event Processor, this chokes the scheduler for minutes.

Why this hurts: we once had a 45-minute gap between a master job finishing and the first dependent job starting. The Event Processor was stuck in a condition re-evaluation loop. The master job's status was SUCCESS. The dependent jobs had their conditions met. But the Event Processor couldn't process the next STARTING event until it finished evaluating all 8,000 dependents. No parallelism. No queue priority. Just a single-threaded condition scan.

Design for this. Batch dependents. Use box jobs to group dependencies. Never put 8,000 jobs on the same condition.

// io.thecodeforge — devops tutorial

// Before: 8,000 dependents on one master — bad
job: daily_payment_export
  condition: p(master_report)

// After: Group via box, reduce condition evaluations
box: payment_pipeline
  condition: p(master_report)

  job: payment_export_region_a
    box: payment_pipeline
    condition: p(box:payment_pipeline)

  job: payment_export_region_b
    box: payment_pipeline
    condition: p(box:payment_pipeline)

// Now master_report finishes -> triggers box success -> dependent box jobs fire
// 10 conditions evaluated instead of 8,000