AutoSys File Watcher: 5 Production Traps & Fixes
- File Watcher = event-driven trigger on file arrival — perfect for unpredictable upstream schedules
- min_file_size default 0 is dangerous. Set to 1024 bytes minimum. Never leave at 0 in production.
- run_window restricts active hours but doesn't filter by file age. Stale files trigger at window open.
- File Watcher (job_type: FW) triggers downstream jobs the moment a file arrives — event-driven, not time-based
- watch_file supports * wildcards. Be specific: /data/*.csv triggers on every CSV, including partial writes
- min_file_size prevents empty or partial file triggers. Set to 1KB minimum — never leave at 0
- run_window restricts active hours. Without it, stale files from last week trigger immediately on restart
- Production trap: upstream writes temp file then renames. Your wildcard matches the temp file. Trigger on incomplete data.
File Watcher — 60-Second Diagnosis
Watcher stuck RUNNING, file exists
ls -la /path/to/watch_file/*.csvautorep -J FW_JOB -q | grep min_file_sizeWatcher triggered on empty file
find /path/to/watch_dir -size 0 -name '*.csv'autorep -J FW_JOB -q | grep min_file_sizeWatcher triggered on stale file
autorep -J FW_JOB -q | grep run_windowls -la /path/to/watch_dir/ | grep 'Mar 18'Watcher not triggering — no file found
ssh agent_host 'ls -la /path/to/watch_file/ 'autoping -m agent_hostProduction Incident
Production Debug GuideWhen your watcher doesn't watch — or watches too much
You can't schedule a job at a fixed time when you don't control when the data arrives. That's the problem File Watcher solves.
A bank sends trade files anytime between 8 AM and 6 PM. You want processing to start the moment the file lands — not poll every 5 minutes and definitely not guess a start time.
But here's what bites people: empty files trigger the watcher. Stale files from yesterday trigger the watcher. The wrong wildcard matches a temp file mid-write. This article covers the production traps that monitoring won't catch.
Defining a File Watcher job
A File Watcher job has job_type: FW (or just 'f'). The key attribute is watch_file — the full path of the file to watch for. The job completes with SUCCESS the moment it finds a file matching the pattern AND the file meets the min_file_size requirement.
Here's the non-obvious part: the File Watcher doesn't 'consume' the file. It just detects it. The file remains on disk. Downstream jobs are responsible for reading, moving, or deleting it. Multiple watchers can't watch the same file — the first one wins.
insert_job: watch_settlement_file job_type: FW watch_file: /data/inbound/settlement_*.csv watch_interval: 60 /* check every 60 seconds */ min_file_size: 512 /* at least 512 bytes (not empty) */ machine: file-landing-server owner: batchuser date_conditions: 1 days_of_week: mon-fri run_window: "07:00 - 23:00" /* only watch during this window */ alarm_if_fail: 1 description: "Watches for daily settlement file from clearing house" insert_job: process_settlement job_type: CMD command: /opt/scripts/process_settlement.sh machine: processing-server-01 owner: batchuser condition: success(watch_settlement_file)
09:32:07 — watch_settlement_file: SUCCESS
09:32:08 — process_settlement: STARTING */
- Watcher doesn't delete, move, or read the file. It just reports existence.
- Multiple watchers on the same file cause a race. First trigger wins.
- The file is still there after the watcher succeeds. Your downstream job must handle it.
- If downstream fails, the file remains. Watcher won't trigger again unless the file changes.
Understanding watch_file wildcards
watch_file supports the * wildcard, which matches any sequence of characters in a filename. The File Watcher triggers as soon as any file matching the pattern appears. This is especially useful when upstream systems include a date in the filename.
The wildcard matches only the filename, not subdirectories. /data/*/file.csv doesn't work. Use separate watchers for different subdirectories.
Critical nuance: The watcher triggers on the FIRST file matching the pattern. If multiple files arrive simultaneously, the watcher triggers on one, succeeds, and the other files are never detected by that watcher instance. Use atomic file naming to control which file triggers.
/* Match any CSV file starting with 'trades_' */ watch_file: /data/inbound/trades_*.csv /* Match any file in the directory (dangerous — matches everything) */ watch_file: /data/inbound/* /* Match files with a date-stamped name pattern */ watch_file: /data/feeds/FEED_*_DONE.txt /* Be specific to avoid triggering on partial/temp files */ watch_file: /data/inbound/FINAL_trades_*.csv /* not temp_trades_*.csv */ /* Safe pattern for atomic writes: write to .tmp, rename to .ready */ /* Watcher watches .ready files only */ watch_file: /data/inbound/*.ready
run_window — limiting when the watcher is active
Without run_window, a File Watcher with date_conditions: 0 runs continuously 24/7. That's usually not what you want — if a stale file from last week is still in the directory when the watcher starts, it triggers immediately on the old file.
run_window restricts the hours during which the File Watcher will detect the file. Outside the window, it won't trigger even if the file is there. When the window opens again, the watcher checks for files — if an old file is still present, it will trigger at window open.
Critical: run_window does NOT prevent the watcher from seeing old files when the window opens. It only restricts when detection is active. File age is not considered.
/* Only watch for the file between 6 AM and 8 PM */ insert_job: watch_eod_file job_type: FW watch_file: /data/eod/eod_positions_*.dat watch_interval: 30 min_file_size: 10240 machine: data-server-01 owner: batchuser run_window: "06:00 - 20:00" /* won't trigger outside these hours */ alarm_if_fail: 1 /* To avoid stale files at window open — add date to watch path */ watch_file: /data/eod/eod_positions_$(date +%Y%m%d).dat /* Or use a wrapper CMD job that checks file timestamp before processing */
Troubleshooting File Watcher jobs
Common File Watcher problems and how to diagnose them:
Problem: File Watcher stays RUNNING after file arrived Check: Is the file smaller than min_file_size? Is the file on the machine specified in the machine attribute? Is the agent on that machine running? Is current time inside run_window?
Problem: File Watcher triggered on wrong file Check: Is your wildcard pattern too broad? Did an old file match? Did a temp file match? Use autorep -J FW_JOB -q to see the exact watch_file pattern.
Problem: File Watcher didn't trigger at all Check: Is the current time within run_window? Is date_conditions set correctly (must be 1 if using start_times)? Is the file in the exact path specified (case-sensitive on Linux)?
Problem: File Watcher triggers but downstream fails saying file is empty Check: min_file_size likely too low. Upstream may have written a zero-byte lock file first.
# Check File Watcher job status autorep -J watch_settlement_file -d # Check the exact watch_file attribute set autorep -J watch_settlement_file -q # Verify the file actually exists on the target machine ssh file-landing-server 'ls -la /data/inbound/settlement_*.csv' # Check agent is running on the watcher machine autoping -m file-landing-server # Check current autosys time (in case timezone matters) autoflags -a | grep -i time # Check if file is below min_file_size ssh file-landing-server 'wc -c /data/inbound/settlement_*.csv'
Status: RU <- still running — file not yet found
watch_file: /data/inbound/settlement_*.csv
min_file_size: 512
run_window: 07:00 - 23:00
File on agent: /data/inbound/settlement_20260319.csv (0 bytes) <- problem: empty file, min_file_size 512
| Attribute | What it does | Default if omitted | Production recommendation |
|---|---|---|---|
| watch_file | File path pattern to watch for | Required — no default | Use specific patterns with date stamps, avoid /* alone |
| watch_interval | How often to check (seconds) | 60 seconds | 30-60 seconds for most cases — lower = more agent load |
| min_file_size | Minimum file size before success | 0 (any size, including empty) | 1024 bytes minimum, 10240 for data files — never 0 |
| run_window | Hours during which watcher is active | No restriction (24/7) | Match expected file arrival window + 2 hours buffer |
| machine | Agent machine where file is located | Required — no default | Must match the server where file physically lands |
🎯 Key Takeaways
- File Watcher = event-driven trigger on file arrival — perfect for unpredictable upstream schedules
- min_file_size default 0 is dangerous. Set to 1024 bytes minimum. Never leave at 0 in production.
- run_window restricts active hours but doesn't filter by file age. Stale files trigger at window open.
- Wildcards trigger on first match only. Multiple simultaneous files? Use a manifest pattern.
- Watcher detects only — downstream must consume (move/delete). Otherwise same file triggers repeatedly.
- Always verify file existence on the agent machine: ssh agent_host 'ls -la'. Your workstation means nothing.
⚠ Common Mistakes to Avoid
Interview Questions on This Topic
- QWhat is an AutoSys File Watcher job and when would you use one?JuniorReveal
- QWhat does the min_file_size attribute do on a File Watcher?JuniorReveal
- QWhat does run_window control on a File Watcher?Mid-levelReveal
- QYour File Watcher triggered prematurely on an incomplete file — what are the possible causes and fixes?SeniorReveal
- QCan a File Watcher job detect files on a remote server?Mid-levelReveal
- QHow does a File Watcher handle multiple files arriving simultaneously?SeniorReveal
Frequently Asked Questions
What is a File Watcher job in AutoSys?
A File Watcher job (job_type: FW) monitors a specified file path pattern on an agent machine. When a matching file appears and meets the minimum size requirement, the job completes with SUCCESS and triggers any dependent jobs.
It's event-driven: processing starts when the file arrives, not at a fixed schedule time.
What is min_file_size in a File Watcher?
min_file_size specifies the minimum file size in bytes that a file must reach before the File Watcher declares SUCCESS. This prevents the watcher from triggering on empty or partially-written files.
Default is 0 (any file). Always set a value larger than what an empty or header-only file would produce — 1024 bytes minimum, 10240 for data files.
Why is my File Watcher triggering on old files?
If a file from a previous day matches your watch_file pattern and run_window isn't set, or the window just opened, the watcher triggers on any matching file regardless of age.
Fix: Use date-stamped watch_file paths (watch_file: /data/feed_$(date +%Y%m%d).csv). Or add downstream file age validation that fails if the file is older than expected.
Can a File Watcher watch for a file on a remote machine?
The File Watcher checks for files locally on the machine specified in the machine attribute. That machine must have an AutoSys agent installed. So effectively, yes — the watcher runs on that remote machine via its agent.
Common mistake: The file lands on serverA but the watcher has machine: serverB. The agent on serverB checks its local filesystem and never finds the file.
What happens if the file never arrives within the run_window?
If the run_window ends without the file arriving, the File Watcher job moves to FAILURE status (if date_conditions: 1 and no other start conditions). This triggers any alarm_if_fail alert.
The downstream dependent jobs will not run until the next scheduling cycle when the watcher is re-triggered (next day, or manually via sendevent).
Does a File Watcher delete or move the file after triggering?
No. The File Watcher only detects the file's existence. It does NOT delete, move, rename, or read the file. The file remains on disk exactly as it was.
Your downstream CMD job must handle file consumption — reading the data, moving the file to /processed, or deleting it after successful processing. Otherwise, the same file will trigger the watcher again on the next run (if the watcher restarts).
Developer and founder of TheCodeForge. I built this site because I was tired of tutorials that explain what to type without explaining why it works. Every article here is written to make concepts actually click.