Simple AWS backups with Snapshots and AMIs

Apr 30, 2011 in AMI • AWS • BACKUPS • EBS
4 min read

{% raw %}

Are you just getting started with Amazon Web Services? It's a great service, but it faces some reliability challenges. This post explains how to set up a simple backup system using Amazon's "snapshot" "machine image" functionalities.

Dependencies

Before you begin, you will need:

Amazon's standard EC2 Command Line Tools
Alestic's ebs-consistent-snapshot program
Tech Kismet's remove-old-snapshots.php script (which must be so named)

AMI Images

Everyone knows that Amazon Instances are ephemeral. When you shut down your server, it's really gone. So the first thing you want to do is create an "Amazon Machine Image" of your server. You can use an AMI to start new instances anytime you like, and the backup is stored on S3 so it is region-independent. Anyone who felt the EBS regional downtime last week is looking for region-independent backup solutions.

So here's a little script I wrote to do a regular backup of your server to AMI:

#!/bin/bash # Variables. Change this to suit the server. Exceptions should be a comma-separated list of directories to exclude from the backup.

SERVER="server-name" #the name of the server you are backing up
UID="1234-1234-1234" # Your Amazon user ID number
CERT="/path/to/your/certfile.pem" # Your Amazon certificate file
KEY="/path/to/your/key.pem" #Your Amazon key file
ACCESSKEY="12345567823SSDFGS6" #your Amazon access key
SECRETKEY="aSDFOUxsfoiusfSDF12SF/Sfi0Asgoiwre" #your amazon secret key
EXCEPTIONS="/ebs,/mnt" #Directories you want to exclude
DESTINATION="/mnt" #Temporary filespace for building the backup itself

ARCH="x86_64" # Is your Instance 32 or 64 bit? DATE=`date +%Y-%m-%d` #today's date # set up nice error handling function error_exit { echo "$1" 1>&2 exit 1 } rm -rf "$DESTINATION"/backup* echo "$(date) Starting instance backup script." echo "creating backup bundle..." if /usr/bin/ec2-bundle-vol -u $UID -c $CERT -k $KEY -p backup-$SERVER-$DATE -r $ARCH -e "$EXCEPTIONS" -d "$DESTINATION"; then echo "SUCCESS" else error_exit "ERROR creating bundle. Exiting..." fi echo "uploading backup bundle..." if /usr/bin/ec2-upload-bundle -b backup -m "$DESTINATION"/backup-$SERVER-$DATE.manifest.xml -a $ACCESSKEY -s $SECRETKEY; then echo "SUCCESS" else error_exit "ERROR uploading bundle. Exiting..." fi echo "registering backup bundle with AWS..." if ec2-register -K $KEY -C $CERT backup/backup-$SERVER-$DATE.manifest.xml; then echo "SUCCESS" else error_exit "ERROR registering bundle. Exiting..." fi echo "deleting backup cache..." if rm -rf "$DESTINATION"/backup*; then echo "SUCCESS" else error_exit "ERROR deleting backup cache. Exiting..." fi echo "backup commands run successfully" exit 0

Run this guy on cron, and you have a regular backup going. The one thing it does not do is prune old backups. Most server setups are fire-and-forget - this is a way of documenting that firing process for next time.

EBS Snapshots

The counterpart to Instances are EBS data stores. These are block devices that provide permanent storage on Amazon's cloud. Here's my backup script for an attached EBS store.

#!/bin/bash # Variables. Change this to suit the server. Mysql User needs access to lock and unlock tables. Number is how many old backup snapshots to keep. SERVER="server-name" #the name of the server you are backing up, for labeling VOLUME="vol-1234567a" #the volume ID that you are backing up MOUNTPOINT="/ebs" #where the volume is mounted MYSQLUSER="root" #a mysql user with access to lock all tables MYSQLPASS="password" #password for the mysql user KEYID=12345799asf8675asf #your Amazon key SECRETKEY=asf765sdg876wrkhalsfa/234@$ # your Amazon secret key NUMBER="15" # How many old backups to keep DATE=`date +%Y-%m-%d` #today's date # set up nice error handling function error_exit { echo "$1" 1>&2 exit 1 } echo "$(date) Starting EBS backup script." echo "creating EBS snapshot..." if /usr/bin/ec2-consistent-snapshot --description "$SERVER backup $DATE" --aws-access-key-id $KEYID --aws-secret-access-key $SECRETKEY --mysql-username $MYSQLUSER --mysql-password $MYSQLPASS --xfs-filesystem "$MOUNTPOINT" $VOLUME; then echo "SUCCESS" else error_exit "ERROR creating snapshot. Exiting..." exit 1 fi if /usr/bin/php /usr/sbin/remove-old-snapshots.php -v $VOLUME -n $NUMBER ; then echo "SUCCESS" else error_exit "ERROR removing old snapshots. Exiting..." exit 1 fi echo "backup commands run successfully" exit 0

This little guy is very useful if you have an EBS device that you use for a running MySQL instance. It will lock MySQL tables briefly, and start the snapshot process from Amazon. This one does prune old backups, you can specify how many you want to keep.

The biggest down side to the EBS snapshot is that it is zone-locked. An EBS snapshot can only be used in the zone where it was created. So in the case of a zone-specific outage, you're SOL.

This is not a proper, region-independent, auto-pruning, tested backup system. This is a set of basic element scripts that you can run on cron, and have some peace of mind while you look into something more robust. But it is a good start, at least!

{% endraw %}