Automatic S3 backups Clark Wilkins, Simplexable 2019.07.02
I want to share a (relatively basic) shell script that solved a really important problem for us. As you may know, we provide warehouse/order management services via mystockd.com and job management via myservicd.com. Part of our services package for both sites is independent databases for each client, and nightly backups.
In times past, I backed up the relatively small client list for INSIGHT to a remote 3TB drive. That's no longer tenable with these new services. I needed something easily extensible and completely cloud-based where our Amazon EC2 servers could backup to our Amazon S3 repositories.
A bit of research into Amazon Command Line Interface tools led to the answer.
#!/bin/bash
# very important to define the home directory and
# where to find the IAM credentials for access
export HOME=/root
AWS_CONFIG_FILE="~/.aws/config"
# add today's source code
now=$(date +"%H.%M.%S")
echo "$now: adding current servicd source code before creating backups"
rm -rf /root/backups/servicd/*
cd /root/backups/servicd/
mkdir source
cp -R /var/www/html/servicd/ source/
# strip out files not specific to the SaaS part of servicd
rm -rf ~/backups/servicd/$today/source/servicd/[long list of files that are on the public side of the site]
# prepare to dump
now=$(date +"%H.%M.%S")
echo "$now: starting backup of servicd clients"
# this is the list of active servicd clients
input="/root/backups/servicd_list"
# run through the list, getting one name at a time
while read -r line
do
now=$(date +"%H.%M.%S")
echo "$now: backup of $line"
# remove prior archives and dumps
rm -rf *.gz
rm -rf *.dmp
# dump databases (add to servicd_list as needed)
pg_dump $line > $line.dmp
# compress the archive
now=$(date +"%H.%M.%S")
echo "$now: compressing backup for $line"
tar -zcvf ~/backups/servicd/$line.gz ~/backups/servicd/
now=$(date +"%H.%M.%S")
echo "$now: ready to backup to S3"
# dump to S3 archive
date=$(date +"%Y.%m.%d")
/root/bin/aws s3 cp /root/backups/servicd/$line.gz s3://backups.myservicd.com/$line/$date.gz
now=$(date +"%H.%M.%S")
echo "$now: $line backup is done"
done < "$input"
now=$(date +"%H.%M.%S")
echo "$now: all backups are done"
Here are the main points:
That's it. We hope this is helpful to others. Regards!