duplicity backup - duplicity-backup.sh

Intro

In part 1, we’ve seen how to generate GPG keys and getting started with duplicity. Part 2 covered moving data off the server using sftp and scheduling the jobs using cron.

In this part, we’re going to use an open source duplicity script which will simplify scripting backups as well as eventually restoring files. Until now, we needed to remember the full command line of the duplicity command. This makes handling fairly difficult. The script has a very good readme page - so i recommend to dig deeper into the different options on your own.

New automation script

In order to get the script, we will be using git. If git --version shows “command not found”, install git with sudo apt-get install git. Next, we need to clone the github repository and copy the configuration-file example to our own configuration.

git clone https://github.com/zertrin/duplicity-backup.sh
cd duplicity-backup.sh
git checkout master
cp duplicity-backup.conf.example duplicity-backup.conf

The script gives us a simple configuration file where all options can be set. The logic is in another file.

Open the configuration file with your favorite editor and modify the variables ROOT, DEST, PASSPHRASE, INCLIST, GPG_ENC_KEY, GPG_SIGN_KEY, STATIC_OPTIONS, LOGDIR.

The relevant content of the files should look as follows:

# Folder to backup
ROOT="/tmp/backupTest/"
# Destination
DEST="sftp://matt@backupserver:1234//home/matt/offsitebackup"

# Password for ssh key
FTP_PASSWORD="SSHTest1!"

INCLIST=( "*" )
EXCLIST=( "" )

#Passphrase for Signing key
PASSPHRASE="TestSig1!"
# Encryption key
GPG_ENC_KEY="3E988E6866B39EE1"
# Signing key
GPG_SIGN_KEY="E24E7891636093DB"

STATIC_OPTIONS="--full-if-older-than 14D --s3-use-new-style"

LOGDIR="/var/log/"
LOG_FILE="duplicity-$(date +%Y-%m-%d_%H-%M).txt"
LOG_FILE_OWNER="root:root"

There are a lot of other options in the configuration - most have sensible defaults (keeping a total of 4 full backups and cleanup older ones, logging, …). Please note that this tutorial is based on the master branch as it was on github on Oct 23rd 2017. The default configuration may change over time as this is a still actively developed project.

After making the above changes, let’s try to run the script. The script has pretty good error-checking - making sure folders exist and the permissions are correct.

./duplicity-backup.sh --config duplicity-backup.conf --full

Depending on how often the backup has been tested before, we may have different outcomes on the cleanup-section. In my case, there were multiple full backups to delete as i tested full backups very often initially.

Now, we’ll replace our crontab-entry we made last time with the new entry:

#old:
#0 1 * * * root /root/backupscript.sh >> /var/log/backupscript.log 2>&1
#new:
0 1 * * * root /root/duplicity-backup.sh/duplicity-backup.sh --config /root/duplicity-backup.sh/duplicity-backup.conf >> /var/log/backupscript.log 2>&1

The script simplifies our life a lot as otherwise we would need to always specify the exact path to the backup location (which can be very frustrating if the remote path is a Amazon S3 bucket or something similar). It would probably be OK for scheduling - but as soon as you need to recover a file, running duplicity-backup.sh [-c config_file] --restore-file backup2/file1 will be a life-changer.

Backup script configuration

Run ./duplicity-backup.sh --config /root/duplicity-backup.sh/duplicity-backup.conf --backup-script to backup the configuration and GPG keys - so the restore-process can be migrated to another server easily. This process will ask you for both key passwords, as well as a backup-password (Yes, the backup will be encrypted as well) and show the decryption-instruction on screen.

Wrapping up

When configuring the backup, always test the backup at least once (best regularly) by restoring to another folder. It’s not sufficient to have a backup-process - it’s equally important to know that the restores will work as well. Also, consider setting up one of the mail options - the script can send mails when the process failed - or each time it ran - so you are notified should something go wrong. Otherwise, as long as nothing happens, backups may run in unnoticed in the background for months - only to discover afterwards that there was a problem and the backup did not run in the last days/weeks/months.

I may cover setting up email notifications in a future blogpost.

In the next part, we will take a look at some of the different cloud-based-options duplicity offers and how to configure them with the duplicity-backup script.