Detecting file system changes on Linux & BSD

Watching the file system for changes comes useful from time to time. For example: checking an FTP/SFTP/rsync directory for newly uploaded files from some external system, looking for new core dumps, and detecting new lines in an error log file. Other use cases might include waiting for a process (or pipeline) to complete its job and save the results to a file, or very crude web server hacking detection.

Both Linux and FreeBSD provide this feature through the inotifywait command.

The command is largely identical on Linux and FreeBSD. It can run in blocking mode (exiting when an event happens) or monitor mode (writing all events to a log file). Many file system events can be filtered such as file open, close, delete, attribute change, modification, new file in a directory etc. See the linked man pages for details. A single invocation can monitor many paths.

I found out that setting a watch on FreeBSD consumes a file descriptor per watched path (the "open files" from ulimit -a). On Linux the limit is set by /proc/sys/fs/inotify/max_user_watches. This means that watching the file system for changes does not scale well. If you need to monitor a million files for changes then maybe you should not architect your system around raw files to start with. 🙂

I wrote a small Bash script that sets up watches on multiple paths and sends me an e-mail when changes are detected, but inotifywait can also be used for all kinds of automations.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

#!/usr/bin/env bash
EMAIL=somewhere@example.com

# Paths to watch
W=""
W="$W $HOME/path1/some_dir/"
W="$W $HOME/path2/single_file.txt"
W="$W $HOME/path3/yet/another/dir/"

PATH_TO_WATCH="$W"
LOG_FILE="$HOME/.logs/inotify.log"

# Kill subprocesses when exiting https://stackoverflow.com/a/2173421
trap "trap - SIGTERM && kill -- -$$" SIGINT SIGTERM EXIT

rm -f $LOG_FILE

# Start the first instance IN THE BACKGROUND in monitor mode and save events to log file
inotifywait \
        --timefmt "%Y-%m-%d %H:%M:%S %Z" \
        --format "%T %w %f %e" -m \
        -o $LOG_FILE \
        -e modify,attrib,close_write,moved_to,moved_from,move,move_self,create,delete,delete_self \
        $PATH_TO_WATCH &

# Let the first inotifywait start and set up the log file
# Yes - this is a race condition
sleep 5

while true; do
        # Watch the log file for changes
        inotifywait -e modify $LOG_FILE #this is blocking
        status=$?
        date
        echo "Log file changed, status $status"
        sleep 15 #give some time in case multiple files have changed, deliver all news at once
        if [ $status -eq 0 ]; then
                cat $LOG_FILE
                echo "Sending email"
                cat $LOG_FILE | mail -s "files changed on $HOSTNAME" $EMAIL
                sleep 21600 #rate limit the email to once every 6 hours
                date
        else
                echo "Not sending mail, status $status"
        fi
        echo "Restarting watch, rearming email notification"
done

The script starts one inotifywait instance in a subprocess. This instance runs in monitor mode (continuously), watches the requested paths, and saves the events to a log file. Then, in a loop, another instance of inotifywait watches (in blocking mode) the log file of the first instance for changes. When it exits, the log file is sent via the mail command to the configured address.

There is a rate limit to avoid accidentally getting a storm of messages in case the file system changes continuously.