AutoSys n_retrys — Why Retry 4 Always Worked

Enterprise batch workflows run overnight when no one's watching. The jobs that matter most — payroll, settlement, reconciliation — are the ones where failures cost the most.

Here's the problem most teams learn the hard way: retries mask flaky scripts until they don't. Box terminators stop bad data from propagating, but only if you put them in the right place. And HA failover? 60-90 seconds feels fast until it's your 2 AM SLA.

This isn't theory. These are the patterns that actually keep workflows alive when things break.

How AutoSys Event Server Event Processor Actually Works

The AutoSys Event Server Event Processor is the core engine that evaluates job dependencies and triggers execution. It polls the event server database for new events—job completions, status changes, or timer expirations—and processes them against a dependency graph. This is a polling-based, not push-based, system: the processor runs in a loop, querying for unprocessed events at a configurable interval (default 10 seconds).

When an event arrives, the processor checks all downstream jobs that depend on it. For each job, it evaluates the dependency condition—typically a Boolean expression of job statuses (SUCCESS, FAILURE, TERMINATED) combined with AND/OR logic. If the condition is met, the processor submits the job to the execution queue. The critical property is that the processor is single-threaded per event server instance, meaning event processing is sequential—no two events are evaluated concurrently for the same job.

Use this processor for any AutoSys environment where job scheduling must respect inter-job dependencies. It matters because it enforces ordering guarantees without requiring custom scripting. In practice, the polling interval and database load are the primary scaling constraints—at high event rates (e.g., 100+ events/second), the processor becomes the bottleneck, and you must tune the poll frequency or shard event servers.

Automatic retry with n_retrys

The simplest fault tolerance mechanism. n_retrys tells AutoSys to automatically rerun a failed job N times before declaring it a final FAILURE. This handles transient failures like brief network blips or temporary database connection issues.

A hidden detail: n_retrys counts retries after the initial attempt. n_retrys: 3 means up to 4 total runs. The retry interval is controlled by the profile setting 'max_exit' default — usually 60 seconds between attempts.

Warning: alarm_if_fail fires only after ALL retries exhaust. If your job succeeds on retry 3, no alarm ever fires. This is good for transient failures but terrible for masking permanent issues.

insert_job: extract_market_data
job_type: CMD
command: /scripts/extract_market.sh
machine: data-server-01
owner: batchuser
date_conditions: 1
days_of_week: all
start_times: "18:00"
n_retrys: 3            /* retry up to 3 times after initial failure = 4 total attempts */
alarm_if_fail: 1       /* alarm only after all retries exhausted */
term_run_time: 45      /* kill if running over 45 minutes */
std_err_file: /logs/autosys/extract_market_data.err

/* To alert on first failure regardless of retries — use separate monitoring */
/* Add a dummy dependency job that detects failure status via autorep */

box_terminator — stopping the box on critical failure

In a BOX with multiple independent jobs, a failure in one job normally leaves other jobs to continue. That's usually what you want — a reporting job failing shouldn't stop the data load.

But sometimes one job's failure should stop everything. If your validation step says 'input data is corrupt', there's zero point running the 50 downstream jobs. They'll just produce garbage.

box_terminator: 1 marks the kill switch. When that job fails, AutoSys immediately terminates the entire box. All pending inner jobs skip to TERMINATED state. The box status becomes FAILURE immediately — no waiting for other jobs to finish.

insert_job: validate_input_data
job_type: CMD
box_name: eod_box
command: /scripts/validate.sh
machine: server01
owner: batch
box_terminator: 1          /* if this fails, the entire box fails immediately */
alarm_if_fail: 1

/* Without box_terminator: other jobs in the box would continue even after validate fails */
/* With box_terminator: box immediately moves to FAILURE, all pending inner jobs skip */

/* Box definition */
insert_job: eod_box
job_type: BOX
owner: batch
date_conditions: 1
start_times: "23:00"

term_run_time — preventing the infinite hang

A job that runs forever is worse than a job that fails. Failing at least triggers alerts and retries. Hanging just blocks everything downstream indefinitely.

term_run_time kills a job after N minutes from its start time. The count begins when the job starts (including retries — each retry resets the timer). When term_run_time expires, AutoSys sends a SIGTERM to the agent. The agent terminates the job process and updates status to TERMINATED.

Crucial difference: TERMINATED is NOT FAILURE. Conditions like success(job) won't trigger on TERMINATED. If you want downstream jobs to run after a timeout, you need condition: status(job) != 'RUNNING' or a custom wrapper script that checks exit codes.

insert_job: nightly_reconcile
job_type: CMD
command: /scripts/reconcile.sh
machine: finance-server
owner: batch
date_conditions: 1
start_times: "23:00"
term_run_time: 390          /* 6.5 hours — kill if still running at 5:30 AM */
run_window: "23:00 - 05:30" /* advisory only — term_run_time does the kill */
alarm_if_fail: 1

/* For downstream jobs that should run even if this times out: */
condition: success(nightly_reconcile) OR status(nightly_reconcile) = 'TERMINATED'

/* Or better: wrapper script */
command: /scripts/reconcile_with_timeout.sh

HA architecture for fault tolerance

At the infrastructure level, AutoSys supports high availability through the dual Event Server architecture. For mission-critical batch environments, this is non-negotiable.

How it works: Primary Event Server handles all writes. Shadow Event Server maintains a real-time replica via database replication (Oracle Data Guard, Sybase Replication, etc.). The Event Processor monitors the primary through heartbeat checks (default 60 seconds).

When the heartbeat fails, the Event Processor promotes the shadow to primary. Total downtime: 60-90 seconds. During this window, running jobs continue unaffected. However, no new jobs start. The Event Processor queues events during failover and processes them once the new primary is online.

Critical nuance: Replication lag is your enemy. If the shadow is 5 minutes behind when the primary fails, you lose 5 minutes of events. Those job completions, status changes, and sendevent calls are gone.

# Check which Event Server is currently primary
autoflags -a | grep -i 'primary\|shadow\|active'

# Verify shadow is in sync
autoflags -a | grep -i 'shadow\|standby'

# Check Event Processor status (should be RUNNING on primary)
chk_auto_up -A

# Check replication lag (Oracle example)
sqlplus autosys_user @check_lag.sql
SELECT APPLIED_LAG FROM V$DATAGUARD_STATS;

# Manual failover (test only)
sendevent -E SWITCH_TO_SHADOW

How the Event Processor Drops Jobs (and Why You Should Care)

The Event Server doesn't process every event. It drops them. Not randomly — by design. When a job definition changes mid-flight, or a manual event comes in while the scheduler is crunching, the processor throttles. It uses a priority queue and a sliding window. If the event rate exceeds the configured max_events_per_second, the processor starts ignoring lower-priority events. That means your 'run now' request might get silently eaten if the system is busy processing a batch of 2000 nightly jobs.

You check syslog. Nothing. You check the event server logs. Silence. You start blaming the network. Stop. Check event_svr_config. The default max_events_per_second is 500. If you're running 2000 jobs with staggered starts, you're hitting that ceiling every 4 seconds. Events get dropped without warning. No error code. No log entry. Just gone.

The fix is straightforward: raise the limit based on your workload, or — better — implement a client-side backoff in your event submission scripts. Never assume the processor took your event. Always verify with autosyslog or event_ack.

// io.thecodeforge — devops tutorial
//
// Verify the current event processing rate limit
// and identify dropped events.

command:
  check_event_processor:
    executable: event_svr_config
    arguments:
      - --show
      - max_events_per_second
    expected_output: 500

log_analysis:
  target: /var/log/autosys/EventProcessor.log
  search_string: "dropping event"
  time_window: "last 15 minutes"
  action: alert_on_match

remediation:
  if event_rate > 450:
    increase_max_events_per_second:
      new_value: 1000
      restart_service: false
      apply_immediately: true

The Hidden Cost of Global Event Filters

Global event filters look like a clean solution. You set a filter to ignore all events from a retired application. Problem disappears. Except now the processor is still parsing every event, matching it against the filter, and discarding it. That's CPU time. That's I/O. That's the event queue growing while your processor burns cycles on discards.

Every event goes through the full pipeline: socket read, parse, authenticate, filter match, priority assign, queue insert. A filter at the end doesn't skip the first three steps. With a global filter matching 30% of events, you're wasting 30% of your processor's throughput on work that produces nothing. Over a 24-hour window, that translates to minutes of latency for legitimate events.

The smarter play: use network-layer filtering or application-level segregation. Block events at the source. If you must use global filters, measure the filter-match CPU time with event_svr_config --stats. If it's above 5% of total CPU, you're burning resources. Drop them upstream.

// io.thecodeforge — devops tutorial
//
// Measure the cost of global event filters
// and recommend removal if wasteful.

statistics:
  collection_interval: 300  # seconds
  metrics:
    - name: event_processor.cpu.time_filter_match
      unit: percent
      threshold: 5
    - name: event_processor.events.filtered_total
      unit: count
    - name: event_processor.events.processed_total
      unit: count

analysis:
  formula: (events_filtered_total / events_processed_total) * 100
  action_on_exceed:
    if percentage > 20:
      recommendation: "Move filtering upstream to application layer. Global filters waste CPU."
      commands:
        - stop: event_svr
        - remove: global_filter_list
        - start: event_svr

Event Processor Syntax: The Declarative Contract

The AutoSys Event Processor uses a declarative YAML syntax to define job behavior, retry policies, and box termination rules. Every job definition must start with insert_job: followed by a unique job name. The job_type: field accepts c, box, or f (command, box, file watcher). Critical attributes include command: for executable paths, machine: for target hosts, and owner: for execution credentials. Control flow is handled via condition: expressions using logical operators (s(jobA) for success, f(jobB) for failure). Box jobs group child jobs with a box_name: attribute; global settings like n_retrys and term_run_time cascade from parent boxes. The alarm_if_fail: boolean triggers alerts. All timestamps use 24-hour format in date_conditions: blocks. This strict syntax prevents silent failures — missing job_type: or malformed conditions cause the Event Processor to reject the job entirely, logging a clear rejection reason to $AUTOUSER/events.log.

// io.thecodeforge — devops tutorial
// Syntax: valid job with conditions
insert_job: data_pipeline_job
job_type: c
command: /app/run_pipeline.sh
machine: prod-db-01
owner: deployer
n_retrys: 2
term_run_time: 300
condition: s(etl_complete) & f(alert_prev)
alarm_if_fail: y
description: "Extract-transform-load step"

Real-World Examples: Retry, Termination & HA in Action

A common production pattern uses n_retrys: 3 and term_run_time: 600 to handle transient failures. Example: a file ingestion job that retries on network blips but aborts after 10 minutes. The Event Processor logs each retry attempt in $AUTOUSER/events.log with a JOB_RETRY event. When box_terminator: y is set on a parent box, a single child failure with alarm_if_fail: y triggers immediate termination of all running siblings — not just future ones. For HA, deploy a minimum of 3 Event Servers behind a load balancer. Each server shares an autosys_ha_file: on NFS (autosys_ha_file: /shared/autosys/ha_state). During failover, the secondary Event Server reads the last processed event from the shared file, resuming exactly where the primary left off. Global event filters (%include for job prefixes, %exclude for machines) reduce noise but accelerate database fragmentation — rotate the $AUTOSYS/events.db weekly to maintain performance.

// io.thecodeforge — devops tutorial
// HA Event Server config with recovery
insert_job: payment_box
job_type: box
box_terminator: y
term_run_time: 900
alarm_if_fail: y
description: "Payment processing group"
---
insert_job: process_transaction
job_type: c
command: /app/process.sh
machine: pay-app-01
n_retrys: 3
condition: s(payment_box)
---
insert_job: notify_failure
job_type: c
command: /app/alert.sh
condition: f(process_transaction)