AutoSys alarm_if_fail Defaults to 0 — Silent Failures

AutoSys has a built-in alarm system that lets you define exactly what events should trigger alerts and who should be notified. Without alarms, your batch jobs could silently fail overnight and nobody would know until users start reporting missing reports in the morning. With well-configured alarms, your team knows within minutes.

But alarms are dangerous. Set them too broadly and your team ignores them (alarm fatigue). Send them to the wrong mailbox and nobody reads them. Default alarm_if_fail: 0 means your critically important job fails every night at 2am and nobody ever hears about it.

By the end you'll know how to set up job failure alarms, email notifications, machine monitoring, and runtime bounds alerts. You'll also know the specific mistakes that cause alarms to be ignored or to never fire at all.

Why AutoSys alarm_if_fail Defaults to 0 — Silent Failures

AutoSys alarm_if_fail is a job attribute that controls whether a failure triggers an alarm notification. When set to 0 (the default), a job that ends in FAILURE status will not generate any alarm — it fails silently. This is the core mechanic: alarm_if_fail=0 means the system treats the failure as a non-event for notification purposes, even though the job's exit code is non-zero.

In practice, alarm_if_fail operates independently of job termination logic. A job can fail, trigger downstream dependencies, and log errors — but if alarm_if_fail=0, no email, SNMP trap, or operator console alert is sent. This is often confused with the job's 'failure' flag; they are not the same. The attribute only controls notification, not job state or rerun behavior.

Use alarm_if_fail=1 for any job where a failure requires human intervention — critical batch processes, data pipelines, or SLA-bound tasks. Leaving it at 0 is acceptable only for jobs where failure is expected or handled by other logic (e.g., cleanup jobs that can fail harmlessly). In production, silent failures from alarm_if_fail=0 are a leading cause of undetected data loss and missed SLAs.

alarm_if_fail — the basic failure alert

The simplest alarm is alarm_if_fail. Set it to 1 on any job, and AutoSys raises an alarm in the Event Server when that job fails. You can then configure alarm actions to send email, page, or invoke a script.

insert_job: critical_eod_job
job_type: CMD
command: /scripts/critical_eod.sh
machine: prod-server-01
owner: batchuser
date_conditions: 1
days_of_week: all
start_times: "22:00"
alarm_if_fail: 1          /* raise alarm if job fails */
max_run_alarm: 60         /* also alarm if still running after 60 minutes */
min_run_alarm: 5          /* alarm if completes in less than 5 minutes (suspicious) */
alarm_if_terminated: 1    /* alarm if job is killed/terminated */

Notification attributes — email on failure

For direct email notification without configuring a separate alarm action, use the notification attributes. These send an email when the job fails.

insert_job: payroll_run
job_type: CMD
command: /scripts/payroll.sh
machine: finance-server
owner: finuser
date_conditions: 1
days_of_week: fri
start_times: "18:00"
alarm_if_fail: 1
/* Email notification on failure */
notification_emailaddress: batch-ops@company.com,payroll-lead@company.com
notification_emailaddress_on_success: payroll-mgr@company.com
notification_msg: "ALERT: AutoSys job %s FAILED on %m at %t. Exit code %x. Check log: /logs/autosys/payroll_run.err"
notification_msg_on_success: "INFO: Payroll run %s completed successfully at %t"

Machine and system alarms

Beyond job-level alarms, AutoSys can alarm on machine events — when an agent goes MISSING or when the Event Processor has issues.

/* Configure machine-level alarms */
update_machine: prod-server-01
max_load: 100
alarm_on_missing: 1    /* alarm when agent goes offline */

/* View active alarms */
# autorep -a              /* show all active alarms */
# sendevent -E ALARM_ACK  /* acknowledge an alarm */

The $SCHEDULER_ON_CALL Variable Death Spiral — Overriding Alarms at Scale

You've got a hundred jobs. They all call $SCHEDULER_ON_CALL when they fail. Fine for a dozen. But when a box job spawns 40 downstream failures in under a minute, everyone gets paged simultaneously. Your team starts ignoring pages. That's how incidents turn into fires.

The root cause? AutoSys evaluates SCHEDULER_ON_CALL globally per job definition. No debouncing. No dedup. If you're overriding this variable globally to a PagerDuty integration, you're bypassing the one layer that could save your on-call: escalation rules.

Instead, use a per-job escalation chain with conditional notification triggers. Set alarm_if_fail to your PagerDuty service ID, then control the noise tiers with minimum failure thresholds. Global variables are for static config, not firehose suppression. If you must override, scope it to a specific box or machine group using the JOB: prefix.

// io.thecodeforge — devops tutorial

// Bad: blind global override that pages everyone
insert_job: finance_end_of_day_box  job_type: BOX
box_name: finance_etl_box
alarm_if_fail: 1
notification: fail_writer
SCHEDULER_ON_CALL: pagerduty_integration_policy

// Good: scoped override with escalation control
insert_job: finance_etl_transaction  job_type: CMD
machine: prod_finance_db_01
command: /scripts/etl/load_transactions.sh
alarm_if_fail: 2
notification: fail_writer
condition: success(finance_etl_box)

// Foreach job in finance_etl_box, override only this job's variable
// Not the whole scheduler scope

The Silent Retry Trap — Why alarm_if_fail Defaults to 1 and max_exit_codes Lie

Here's a classic: you set max_retry 3, max_exit_codes 0, and alarm_if_fail 1. You think it works. One day a job fails with exit code 1 on its third retry. AutoSys retries 3 times, each one fails, then it sends an alarm. Your on-call gets paged 45 minutes after the first failure. Why? Because alarm_if_fail triggers AFTER the last retry — every attempt succeeds? Every failure counts.

max_exit_codes only works for the final attempt. If your job fails on retry 2 with exit code 1, AutoSys treats that as a success if 1 is in your max_exit_codes list. But if it fails on retry 3 with exit code 5 (not in the list), alarm_if_fail fires. You've effectively masked failures for 2/3 of the retries, then screamed about the last one. That's not alerting. That's lying to yourself.

Set alarm_if_fail to 0 if your max_exit_codes list is exhaustive and retry logic handles the noise. Otherwise, keep it at 1 and audit your runbook. You want to know about the retry that failed, then the retry that succeeded — not the silent one that never happened.

// io.thecodeforge — devops tutorial

// This job masks 2 failures, screams on the 3rd
insert_job: api_daily_reconcile  job_type: CMD
machine: prod_api_01
command: /apps/api/scripts/reconcile.sh
alarm_if_fail: 1
max_retry: 3
max_exit_codes: 0,1,2

// What actually happens:
// Attempt 1: exit 1 (masked, counts as success because 1 in max_exit_codes)
// Attempt 2: exit 1 (masked again)
// Attempt 3: exit 5 (NOT in max_exit_codes, alarm fires)

// Better: be transparent about failure
// Set alarm_if_fail 0 and use a separate monitor for retry patterns