A customer had an issue with s3fs dying like so:
[702849.587172] Out of memory: Kill process 854 (s3fs) score 65 or sacrifice child [702849.587184] Killed process 854 (s3fs) total-vm:1148284kB, anon-rss:252944kB, file-rss:0kB
Our solution is to use monit to auto-remount the s3fs directory if it dies. In this example, it is the wordpress uploads folder we are monitoring as it's used to share files between two load balanced servers (not the best solution, but it works).
# apt-get install monit
2.Enable shell access to monit by editing /etc/monit/monitrc and uncommenting the following:
set httpd port 2812 and use address localhost # only accept connection from localhost allow localhost
# service monit restart
4.Create some scripts
# mkdir /etc/monit/scripts/
#!/bin/sh /usr/bin/s3fs bucketname -o allow_other /home/user/public_html/wp-content/uploads
#!/bin/sh fusermount -uz /home/user/public_html/wp-content/uploads
Check file s3fs with path /home/user/public_html/wp-content/uploads/monit-test-dont-delete.txt start program = "/etc/monit/scripts/start-s3fs.sh" stop program = "/etc/monit/scripts/stop-s3fs.sh" if does not exist then restart if 5 restarts WITHIN 5 cycles then timeout
monit test file, don't delete
5.Now you can restart monit, manually run /etc/monit/scripts/stop-s3fs.sh to trigger a stop, watch
mount and /var/log/monit.log to ensure your monit service detects the directory being unmounted and ensure your script is run to remount it.
# service monit restart # /etc/monit/scripts/stop-s3fs.sh # tail -f /var/log/monit.log ... # mount