AutoSys FORCE_STARTJOB — Condition Bypass Corrupts Data

Production AutoSys environments need manual intervention. Jobs hang. Downstream dependencies get stuck. A fix deploys and you need to rerun a failed job at 2 AM.

Knowing the exact sendevent command is table stakes. Knowing what happens after — that's the senior engineer difference.

FORCE_STARTJOB bypasses conditions. KILLJOB leaves downstream jobs waiting. RESTART is for retries, not first runs. This article covers the side effects that incident post-mortems reveal.

How AutoSys FORCE_STARTJOB Breaks Job Dependencies

FORCE_STARTJOB is an AutoSys command that starts a job immediately, bypassing all upstream conditions, box dependencies, and calendar rules. It ignores the job's defined starting conditions entirely — no waiting for predecessor success, no checking for box status, no respecting time windows. The job runs as if all conditions are met, regardless of reality.

When you issue sendevent -E FORCE_STARTJOB -J job_name, AutoSys sets the job's status to STARTING and launches it. The job's exit code still updates its status (SUCCESS/FAILURE), but the event log records the forced start. Downstream jobs that depend on this job will see the status change and may trigger, even though the data the job processed might be stale or incomplete because upstream dependencies were skipped.

Use FORCE_STARTJOB only when you have manually verified that all upstream data is ready and consistent — typically during disaster recovery or after manual data reconciliation. In production, teams often reach for it to unstick a stalled pipeline, but the bypass corrupts data integrity because downstream consumers see a success signal without the actual dependency chain being satisfied.

FORCE_STARTJOB — bypassing all conditions

FORCE_STARTJOB immediately starts a job regardless of its date_conditions, start_times, or condition dependencies. It's the 'run it now, no questions asked' command.

Critical nuance: FORCE_STARTJOB bypasses EVERYTHING. Not just the schedule. Not just the time gates. Also any condition: success(other_job) dependencies. The job runs even if its upstream dependencies never ran or failed.

This is the most dangerous sendevent command. Use it only when you fully understand what conditions exist on the job.

# Force start a single job immediately
sendevent -E FORCE_STARTJOB -J daily_report

# Force start a BOX (starts the box, inner jobs still follow their conditions)
sendevent -E FORCE_STARTJOB -J eod_processing_box

# Force start on a specific date (run it as if it were that date)
sendevent -E FORCE_STARTJOB -J daily_report -q 20260319

# Check what conditions you're about to bypass — ALWAYS do this first
autorep -J daily_report -q | grep condition

# Check current status
autorep -J daily_report

KILLJOB — terminating a running job

KILLJOB sends a termination signal to the process running on the agent machine. The job moves to TERMINATED status. Any downstream jobs waiting on success() of this job will not start.

Critical nuance: TERMINATED is NOT FAILURE. It's a separate status. A job that's killed doesn't trigger success() conditions, but it also doesn't trigger failure() conditions unless you explicitly check for TERMINATED.

After KILLJOB, the process receives SIGTERM. Well-behaved processes can catch this and clean up. Hung processes may need SIGKILL (AutoSys handles this escalation after a timeout, typically 30 seconds).

# Kill a running job
sendevent -E KILLJOB -J hung_etl_job

# After killing, check status
autorep -J hung_etl_job

# Check if any downstream jobs are blocked
sendevent -E LIST_DEPENDENTS -J hung_etl_job

# If you need downstream jobs to proceed after kill:
# First kill the job, then manually mark it as success
sendevent -E KILLJOB -J hung_etl_job
sendevent -E CHANGE_STATUS -J hung_etl_job -s SUCCESS

# Check downstream jobs are now unblocked
autorep -J downstream_job

RESTART — retrying a failed job

The RESTART event tells AutoSys to rerun a job that is in FAILURE or TERMINATED status. It's cleaner than FORCE_STARTJOB for rerunning failed jobs because it signals intent as a retry in audit logs.

Key difference from FORCE_STARTJOB: RESTART works only on FAILURE or TERMINATED jobs. FORCE_STARTJOB works on any non-running state. RESTART also respects that this is a retry — some AutoSys configurations treat retries differently for alerting purposes.

RESTART does NOT bypass conditions. The job still needs its start conditions satisfied (unless they were the reason it failed — then you have a cycle).

# Restart a failed job after fixing the root cause
sendevent -E RESTART -J failed_extract_job

# Before restarting, check WHY it failed
autorep -J failed_extract_job -L 10

# Check if the fix actually worked (test mode)
autorep -J failed_extract_job -q | grep command
# Manually run the command on the agent to verify

# Pattern: check for failures and restart them (for transient failures only)
autorep -J % -s FA | awk '{print $1}' | while read job; do
  echo "Restarting failed job: $job"
  sendevent -E RESTART -J "$job"
done

# For TERMINATED jobs (killed), RESTART also works
sendevent -E RESTART -J killed_job

CHANGE_STATUS — manual status override

CHANGE_STATUS is the nuclear option. It manually sets a job's status in the Event Server without running anything. This is how you unblock workflows when a job can't or shouldn't be rerun.

Most common use: A job was killed (KILLJOB) but the work was already done. No point rerunning. Set its status to SUCCESS to fire downstream success() conditions.

Risks: CHANGE_STATUS bypasses ALL validation. No agent communication. No process verification. You're telling AutoSys 'trust me, this job is in this state'. If you're wrong, downstream processes run on incorrect assumptions.

# After a KILLJOB, mark it as SUCCESS to unblock downstream
sendevent -E CHANGE_STATUS -J hung_job -s SUCCESS

# Manually mark a job as FAILURE (e.g., if you see it's going to fail)
sendevent -E CHANGE_STATUS -J running_job -s FAILURE

# Mark a job as INACTIVE to prevent it from running
sendevent -E CHANGE_STATUS -J job_on_hold -s INACTIVE

# Verify the status change took effect
autorep -J hung_job -q | grep status

# Dangerous: Mark a job as SUCCESS that never ran
# Only do this if you have verified the work was done elsewhere
sendevent -E CHANGE_STATUS -J never_ran_job -s SUCCESS

AutoSys Job Recovery: The Incident Report You Weren't Expecting

You force-started a job at 3 AM to unblock the batch pipeline. By 5 AM, five downstream jobs had silently failed because their box conditions never evaluated. The scheduler saw the job as a manual override and skipped all the dependency checks you'd spent weeks building. The production incident report won't mention your name, but your teammates will remember.

When you use FORCE_STARTJOB, AutoSys flags the job with a special attribute that tells the scheduler: "This run is a dirty override — don't trust its completion for dependency resolution." The job runs, exits zero, but its box conditions and success codes are ignored by downstream jobs. The system treats it like a ghost run. That's why intelligent operators build a recovery job — a wrapper that checks the job's exit status and manually updates the downstream conditions using CHANGE_STATUS after verifying the job actually did its work.

The recovery pattern is simple: after the force start completes, run a secondary process that validates the output (file landed, DB updated, API responded), then explicitly sets the condition in the downstream box. Your control room will thank you when the morning batch doesn't implode.

// io.thecodeforge — devops tutorial

// Recovery job to fix downstream dependencies after FORCE_STARTJOB
// Run this immediately after the forced job completes

insert_job: RECVRY_JOB_PAYMENT_PROCESSOR
job_type: CMD
command: /opt/autosys/recovery/validate_and_set_condition.sh
machine: payment-prod-01
description: "validate forced job output then set SUCCESS flag in box"
conditions: s(PAYMENT_PROCESSOR_DUMP)  # start when forced job finishes
start_times: "00:00"  # no time restriction — triggered by dependency
daily_run: n
timezone: America/New_York
profile: /home/autosys/.profile

KILLJOB Deep Cut: Graceful vs. Slaughter Mode

Standard KILLJOB sends SIGTERM. That's the polite wave that says "please finish your transaction and exit." But when you've got a runaway Java process chewing through 16 GB of heap and ignoring polite requests, you need the nuclear option. AutoSys gives you that with KILLJOB /SIGKILL = 1 — but the documentation hides the consequences.

When you send SIGKILL, AutoSys immediately marks the job as KILLED in the scheduler. The process might still be alive for a few seconds before the OS terminates it. But here's the trap: if your job had open file handles (database connections, temporary files, NFS mounts), those get orphaned. The DB connection pool will leak. Temp files stay behind and fill up disk. NFS mounts stay locked.

Senior engineers build a two-stage kill procedure into their runbooks: first KILLJOB (SIGTERM), wait 30 seconds, check autorep -j JOBNAME -w 10 for the KILLED status. If it's still RUNNING, then fire KILLJOB /SIGKILL = 1. After that, run a cleanup script that checks for orphaned temp files by PID, kills any residual processes by PGID, and logs the incident with a timestamp for your SRE dashboard. Do this right once, and the 2 AM page becomes a 5-minute fix instead of a 3-hour war room.

// io.thecodeforge — devops tutorial

// Two-stage kill procedure for production jobs
// Stage 1: polite kill
insert_job: KILL_PAYMENT_BATCH
job_type: CMD
command: |
  autorep -j PAYMENT_BATCH -w 10 | grep RUNNING &&
  sendevent -E KILLJOB -J PAYMENT_BATCH &&
  echo "SIGTERM sent at $(date)" >> /tmp/kill_log_
box_name: BOX_WEBHOOKS_PROCESSING
description: "two-stage kill with cleanup"

// Stage 2: run after 30 sec delay
insert_job: FORCE_KILL_PAYMENT_BATCH
job_type: CMD
command: |
  sleep 30
  if autorep -j PAYMENT_BATCH -w 10 | grep -q RUNNING; then
    sendevent -E KILLJOB -J PAYMENT_BATCH /SIGKILL=1
    echo "SIGKILL sent at $(date)" >> /tmp/kill_log_
    kill -9 $(ps -ef | grep payment_batch | awk '{print $2}') 2>/dev/null
  fi
  /opt/autosys/cleanup/orphaned_files.sh -j PAYMENT_BATCH
box_name: BOX_WEBHOOKS_PROCESSING