File Watcher Jobs in AutoSys — Event-Driven File Monitoring
- File Watcher jobs (job_type: FW) watch for a file to appear and trigger dependent jobs when found — perfect for event-driven data processing
- min_file_size protects against triggering on empty or partially-written files
- run_window restricts when the watcher is active, preventing triggers on stale files outside business hours
File Watcher jobs solve a real and common problem: you can't schedule data processing at a fixed time when you don't control when the data arrives. An upstream bank might send you a trade file anytime between 8 AM and 6 PM. A File Watcher lets you trigger processing the moment the file arrives rather than polling on a schedule.
This article covers everything about AutoSys File Watcher jobs — definition, key attributes, wildcards, run_window, and the gotchas that bite people in production.
Defining a File Watcher job
A File Watcher job has job_type: FW (or just 'f'). The key attribute is watch_file — the full path of the file to watch for. The job completes with SUCCESS the moment it finds a file matching the pattern AND the file meets the min_file_size requirement.
insert_job: watch_settlement_file job_type: FW watch_file: /data/inbound/settlement_*.csv watch_interval: 60 /* check every 60 seconds */ min_file_size: 512 /* at least 512 bytes (not empty) */ machine: file-landing-server owner: batchuser date_conditions: 1 days_of_week: mon-fri run_window: "07:00 - 23:00" /* only watch during this window */ alarm_if_fail: 1 description: "Watches for daily settlement file from clearing house" insert_job: process_settlement job_type: CMD command: /opt/scripts/process_settlement.sh machine: processing-server-01 owner: batchuser condition: success(watch_settlement_file)
09:32:07 — watch_settlement_file: SUCCESS
09:32:08 — process_settlement: STARTING */
Understanding watch_file wildcards
watch_file supports the * wildcard, which matches any sequence of characters in a filename. The File Watcher triggers as soon as any file matching the pattern appears. This is especially useful when upstream systems include a date in the filename.
/* Match any CSV file starting with 'trades_' */ watch_file: /data/inbound/trades_*.csv /* Match any file in the directory (dangerous — matches everything) */ watch_file: /data/inbound/* /* Match files with a date-stamped name pattern */ watch_file: /data/feeds/FEED_*_DONE.txt /* Be specific to avoid triggering on partial/temp files */ watch_file: /data/inbound/FINAL_trades_*.csv /* not temp_trades_*.csv */
run_window — limiting when the watcher is active
Without run_window, a File Watcher with date_conditions: 0 runs continuously 24/7. That's usually not what you want — if a stale file from last week is still in the directory when the watcher starts, it triggers immediately on the old file.
run_window restricts the hours during which the File Watcher will detect the file. Outside the window, it won't trigger even if the file is there.
/* Only watch for the file between 6 AM and 8 PM */ insert_job: watch_eod_file job_type: FW watch_file: /data/eod/eod_positions_*.dat watch_interval: 30 min_file_size: 10240 machine: data-server-01 owner: batchuser run_window: "06:00 - 20:00" /* won't trigger outside these hours */ alarm_if_fail: 1
Troubleshooting File Watcher jobs
Common File Watcher problems and how to diagnose them:
Problem: File Watcher stays RUNNING after file arrived Check: Is the file smaller than min_file_size? Is the file on the machine specified in the machine attribute? Is the agent on that machine running?
Problem: File Watcher triggered on wrong file Check: Is your wildcard pattern too broad? Did an old file match? Consider adding a date-specific element to the watch_file path.
Problem: File Watcher didn't trigger at all Check: Is the current time within run_window? Is date_conditions set correctly? Is the file in the exact path specified (case-sensitive on Linux)?
# Check File Watcher job status autorep -J watch_settlement_file -d # Check the exact watch_file attribute set autorep -J watch_settlement_file -q # Verify the file actually exists on the target machine ls -la /data/inbound/settlement_*.csv # Check agent is running on the watcher machine autoping -m file-landing-server # Check current autosys time (in case timezone matters) autoflags -a | grep -i time
Status: RU <- still running — file not yet found
watch_file: /data/inbound/settlement_*.csv
min_file_size: 512
run_window: 07:00 - 23:00
| Attribute | What it does | Default if omitted |
|---|---|---|
| watch_file | File path pattern to watch for | Required — no default |
| watch_interval | How often to check (seconds) | 60 seconds |
| min_file_size | Minimum file size before success | 0 (any size, including empty) |
| run_window | Hours during which watcher is active | No restriction (24/7) |
| machine | Agent machine where file is located | Required — no default |
🎯 Key Takeaways
- File Watcher jobs (job_type: FW) watch for a file to appear and trigger dependent jobs when found — perfect for event-driven data processing
- min_file_size protects against triggering on empty or partially-written files
- run_window restricts when the watcher is active, preventing triggers on stale files outside business hours
- The file must be on the machine specified in the machine attribute — the agent on that machine does the local file check
⚠ Common Mistakes to Avoid
- ✕Setting min_file_size: 0 (or not setting it) — the watcher triggers on empty or partial files written by upstream systems
- ✕Pointing watch_file to a different machine than where the file actually lands — the agent on the watch machine checks for the file locally
- ✕Using overly broad wildcards that match temp files or old files
- ✕Not setting run_window — the watcher triggers on stale files from previous days when it restarts
- ✕Forgetting that wildcards only match the filename, not subdirectories
Interview Questions on This Topic
- QWhat is an AutoSys File Watcher job and when would you use one?
- QWhat does the min_file_size attribute do on a File Watcher?
- QWhat does run_window control on a File Watcher?
- QYour File Watcher triggered prematurely on an incomplete file — what are the possible causes and fixes?
- QCan a File Watcher job detect files on a remote server?
Frequently Asked Questions
What is a File Watcher job in AutoSys?
A File Watcher job (job_type: FW) monitors a specified file path pattern on an agent machine. When a matching file appears and meets the minimum size requirement, the job completes with SUCCESS and triggers any dependent jobs.
What is min_file_size in a File Watcher?
min_file_size specifies the minimum file size in bytes that a file must reach before the File Watcher declares SUCCESS. This prevents the watcher from triggering on empty or partially-written files. Always set a value larger than what an empty or header-only file would produce.
Why is my File Watcher triggering on old files?
If a file from a previous day matches your watch_file pattern and run_window isn't set, the watcher may trigger on the old file when it starts. Use run_window to restrict active hours, and consider having upstream systems write files to a date-specific subdirectory.
Can a File Watcher watch for a file on a remote machine?
The File Watcher checks for the file locally on the machine specified in the machine attribute. That machine must have an AutoSys agent installed. So effectively, yes — the watcher runs on a remote machine via its agent.
What happens if the file never arrives within the run_window?
If the run_window ends without the file arriving, the File Watcher job moves to FAILURE status, which triggers any alarm_if_fail alert. The downstream dependent jobs will not run until the next scheduling cycle when the watcher is re-triggered.
Developer and founder of TheCodeForge. I built this site because I was tired of tutorials that explain what to type without explaining why it works. Every article here is written to make concepts actually click.