Use monit to monitor a s3fs mount

A customer had an issue with s3fs dying like so:

[702849.587172] Out of memory: Kill process 854 (s3fs) score 65 or sacrifice child
[702849.587184] Killed process 854 (s3fs) total-vm:1148284kB, anon-rss:252944kB, file-rss:0kB

Our solution is to use monit to auto-remount the s3fs directory if it dies. In this example, it is the wordpress uploads folder we are monitoring as it's used to share files between two load balanced servers (not the best solution, but it works).

1.Install monit:

# apt-get install monit

2.Enable shell access to monit by editing /etc/monit/monitrc and uncommenting the following:

 set httpd port 2812 and
    use address localhost  # only accept connection from localhost
    allow localhost 

3.Restart monit:

# service monit restart

4.Create some scripts

# mkdir /etc/monit/scripts/

/etc/monit/scripts/start-s3fs.sh:

#!/bin/sh
/usr/bin/s3fs bucketname -o allow_other /home/user/public_html/wp-content/uploads

/etc/monit/scripts/stop-s3fs.sh:

#!/bin/sh
fusermount -uz /home/user/public_html/wp-content/uploads

/etc/monit/conf.d/s3fs:

Check file s3fs with path /home/user/public_html/wp-content/uploads/monit-test-dont-delete.txt
   start program = "/etc/monit/scripts/start-s3fs.sh" 
   stop program = "/etc/monit/scripts/stop-s3fs.sh" 
   if does not exist then restart 
   if 5 restarts WITHIN 5 cycles then timeout

/home/user/public_html/wp-content/uploads/monit-test-dont-delete.txt:

monit test file, don't delete

5.Now you can restart monit, manually run /etc/monit/scripts/stop-s3fs.sh to trigger a stop, watch mount and /var/log/monit.log to ensure your monit service detects the directory being unmounted and ensure your script is run to remount it.

# service monit restart
# /etc/monit/scripts/stop-s3fs.sh
# tail -f /var/log/monit.log
...
# mount