duplicity backup - getting started
Intro
A basic rule for Backups is the 3-2-1 rule: 3 total copies of the data - on 2 different mediums, at least 1 offsite.
In the following multi-part tutorial I will show how to generate fairly secure backups with duplicity following the above strategy.
Note: My test-system is debian-based - however the commands should also work with most other Linux systems by replacing the name of the package-manager to the system-default. The tutorial will assume that you have root access on the system. Most steps are also possible as regular user - however full system backups are not possible in this case.
About duplicity
Duplicity is a python-based backup-program which has multiple backends. It uses encryption by default and handles the gpg encryption for us. The signature-key defaults to the encryption-key - in that case, the encryption-key is used for signing as well. I strongly recommend to create a separate Keypair - the following will assume you will do the same.
Encryption and keys
Duplicity is able use 2 different methods of encryption.
- symmetric encryption
- asymmetric encryption
This post will focus on asymmetric encryption as this is considered more secure than symmetric encryption (it uses a different key for encryption and decryption). Explaining the cryptographic details would span a whole book, so I’ll leave further research to the interested reader as there are many good resources on this topic available.
To ideally protect our backup, we will be generating 2 GPG key-pairs:
- encryption key pair (confidentiality)
- signing key pair (integrity)
While backing up, the public key of the encryption key pair is used to encrypt the data, while the private key of the signing key pair will be used to ensure the integrity of the backup.
We will use separate keys as the pass-phrase for the signing key needs to be available when running the backup (which is normally a script or scheduled job).
Let’s first make sure we have enough entropy for the key generation.
generating keys - entropy
Note: the key-generation may take a while, depending on the available entropy in the system (check using cat /proc/sys/kernel/random/entropy_avail).
On a fresh test-system I faced this problem, while on my regular PC i had enough entropy available.
The following should be run in a separate session and generate enough entropy
apt-get install rng-tools
rngd -f -r /dev/urandom
Generating the keys
For this tutorial, I will use the passwords TestEnc1! for the Encryption key (named duplicity_enc), and TestSig1! for the signing key (named duplicity_sign). In reality, a random, secure password/pass phrase should be used (this passwords have multiple problems - please never use such passwords for anything). Also, please consider using 4096 bit keysize - this will increase security another bit (i did stick with defaults for this tutorial - but my production implementation does use 4096 bit keysizes).
We will need to run the following command twice, once for the encryption key pair and once for the signing key pair. Make sure to use different passwords, otherwise a single key could be used as well.
Note that the signing-key needs to be generated first.
gpg --full-generate-key
sample output:
gpg (GnuPG) 2.1.18; Copyright (C) 2017 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Please select what kind of key you want:
(1) RSA and RSA (default)
(2) DSA and Elgamal
(3) DSA (sign only)
(4) RSA (sign only)
Your selection?
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048)
Requested keysize is 2048 bits
Please specify how long the key should be valid.
0 = key does not expire
<n> = key expires in n days
<n>w = key expires in n weeks
<n>m = key expires in n months
<n>y = key expires in n years
Key is valid for? (0)
Key does not expire at all
Is this correct? (y/N) y
GnuPG needs to construct a user ID to identify your key.
Real name: duplicity_sign
Email address:
Comment: Test key for duplicity signing
You selected this USER-ID:
"duplicity_sign (Test key for duplicity signing)"
Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: key E24E7891636093DB marked as ultimately trusted
gpg: revocation certificate stored as '/root/.gnupg/openpgp-revocs.d/5BA6FC44F5C7FBF5CCB62A43E24E7891636093DB.rev'
public and secret key created and signed.
pub rsa2048 2017-10-15 [SC]
5BA6FC44F5C7FBF5CCB62A43E24E7891636093DB
uid duplicity_sign (Test key for duplicity signing)
sub rsa2048 2017-10-15 [E]
After generating both keys, the output of gpg --list-keys --keyid-format long should look similar to mine:
/root/.gnupg/pubring.kbx
-----------------------------
pub rsa2048/E24E7891636093DB 2017-10-15 [SC]
5BA6FC44F5C7FBF5CCB62A43E24E7891636093DB
uid [ultimate] duplicity_sign (Test key for duplicity signing)
sub rsa2048/1118A69B4088EC29 2017-10-15 [E]
pub rsa2048/3E988E6866B39EE1 2017-10-15 [SC]
1FD28DA009BBEE82A543B2CC3E988E6866B39EE1
uid [ultimate] duplicity_enc (test key for duplicity encryption)
sub rsa2048/74DCEA71B0217B9B 2017-10-15 [E]
The interesting part is E24E7891636093DB and 1118A69B4088EC29, which contains our key-id’s. The key-id’s will be unique for everyone running though this tutorial - the keys must be replaced with your own for every step they appear in.
Next, we will need to sign the encryption key.
gpg --sign-key duplicity_enc
output:
sec rsa2048/3E988E6866B39EE1
created: 2017-10-15 expires: never usage: SC
trust: ultimate validity: ultimate
ssb rsa2048/74DCEA71B0217B9B
created: 2017-10-15 expires: never usage: E
[ultimate] (1). duplicity_enc (test key for duplicity encryption)
sec rsa2048/3E988E6866B39EE1
created: 2017-10-15 expires: never usage: SC
trust: ultimate validity: ultimate
Primary key fingerprint: 1FD2 8DA0 09BB EE82 A543 B2CC 3E98 8E68 66B3 9EE1
duplicity_enc (test key for duplicity encryption)
Are you sure that you want to sign this key with your
key "duplicity_sign (Test key for duplicity signing)" (E24E7891636093DB)
Really sign? (y/N) y
backup GPG keys
Export public keys and store them in a safe location (if you loose them you cannot get your files back).
# gpg --armor --export -a [keyid] > [public_key_filename].key
gpg --armor --export -a 5BA6FC44F5C7FBF5CCB62A43E24E7891636093DB > duplicity_sign_public.key
gpg --armor --export -a 1FD28DA009BBEE82A543B2CC3E988E6866B39EE1 > duplicity_enc_public.key
Export public keys and store them in a safe location. This will ask you for the passwords/pass phrases provided above.
# gpg --armor --export-secret-keys -a [keyid] > [private_key_filename].key
gpg --armor --export-secret-keys -a 5BA6FC44F5C7FBF5CCB62A43E24E7891636093DB > duplicity_sign_private.key
gpg --armor --export-secret-keys -a 1FD28DA009BBEE82A543B2CC3E988E6866B39EE1 > duplicity_enc_private.key
Details about the gpg key restore process may be shared in a later post.
install duplicity
apt-get install duplicity
Now let’s create a few test folders with 5 files in each folder.
mkdir backupTest
cd backupTest
mkdir backup{1..10}
touch backup{1..10}/file{1..5}
backupTest
|-- backup1
|-- backup1/file1
|-- backup1/file2
|-- backup1/file3
|-- backup2/file1
...
first test
To start, we’ll backup the folder /tmp/backupTest to /tmp/backup. This will ask for the password to the signing key (we are ready to script the backup - let’s test things manually for now).
# duplicity full --encrypt-key [encryptkey] --sign-key [signkey] [folder] file://[backupfolder]
duplicity full --encrypt-key 3E988E6866B39EE1 --sign-key E24E7891636093DB /tmp/backupTest file:///tmp/backup
Note that the target-folder has 3 “/” - this is because I’m using absolute paths here.
To verify the backup run duplicity verify (Target folder and folder are reversed).
This will ask for the password of our encryption key.
# duplicity verify --compare-data --encrypt-key [encryptkey] --sign-key [signkey] file://[backupfolder] [folder]
duplicity verify --compare-data --encrypt-key 3E988E6866B39EE1 --sign-key E24E7891636093DB file:///tmp/backup /tmp/backupTest
--compare-data compares the actual content of the files, not just the files themselves. For remote connections and big backups this is not recommended.
Now, lets change a file echo test > backup2/file2 and rerun the above verify command.
Now the output should show a difference.
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Sun Oct 15 20:14:03 2017
GnuPG passphrase for decryption:
Difference found: File backup2/file2 has mtime Sun Oct 15 20:15:15 2017, expected Sun Oct 15 13:10:53 2017
Verify complete: 61 files compared, 1 difference found.
The 1 difference found highlights the difference we just created. Running the incremental sync again will remove this difference again.
# duplicity incr --encrypt-key [encryptkey] --sign-key [signkey] [folder] file://[backupfolder]
duplicity incr --encrypt-key 3E988E6866B39EE1 --sign-key E24E7891636093DB /tmp/backupTest file:///tmp/backup
output:
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Sun Oct 15 20:14:03 2017
GnuPG passphrase for decryption:
GnuPG passphrase for signing key:
--------------[ Backup Statistics ]--------------
StartTime 1508091392.03 (Sun Oct 15 20:16:32 2017)
EndTime 1508091392.11 (Sun Oct 15 20:16:32 2017)
ElapsedTime 0.07 (0.07 seconds)
SourceFiles 61
SourceFileSize 45086 (44.0 KB)
NewFiles 0
NewFileSize 0 (0 bytes)duplicity-backup.conf_mine
DeletedFiles 0
ChangedFiles 1
ChangedFileSize 10 (10 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 1
RawDeltaSize 15 (15 bytes)
TotalDestinationSizeChange 833 (833 bytes)
Errors 0
-------------------------------------------------
Now we have one full backup - and one additional partial backup.
looking into our backup-folder we see the following:
-rw------- 1 root root 1758 Oct 15 20:14 duplicity-full-signatures.20171015T181403Z.sigtar.gpg
-rw------- 1 root root 1001 Oct 15 20:14 duplicity-full.20171015T181403Z.manifest.gpg
-rw------- 1 root root 1527 Oct 15 20:14 duplicity-full.20171015T181403Z.vol1.difftar.gpg
-rw------- 1 root root 823 Oct 15 20:16 duplicity-inc.20171015T181403Z.to.20171015T181625Z.manifest.gpg
-rw------- 1 root root 833 Oct 15 20:16 duplicity-inc.20171015T181403Z.to.20171015T181625Z.vol1.difftar.gpg
-rw------- 1 root root 846 Oct 15 20:16 duplicity-new-signatures.20171015T181403Z.to.20171015T181625Z.sigtar.gpg
We see that we have one full sync - and one incremental sync.
restore
Testing the backup (aka restoring) is important to be sure the backups are working. Restoring with duplicity is as simple as inverting the backup-command (without full / incr keyword) and supplying the password to the encryption key. For completeness, we will include the restore-keyword to make the intent obvious.
My restore will go to the path /tmp/test2. Restoring to a non-empty path will result in the message “Will not overwrite.”.
# duplicity restore --encrypt-key [encryptkey] --sign-key [signkey] file://[backupfolder] [folder]
duplicity restore --encrypt-key 3E988E6866B39EE1 --sign-key E24E7891636093DB file:///tmp/backup /tmp/test2
This folder now contains the same content than /tmp/backupTest. The restore-process also restores file permissions - which is very handy for full system backups.
Final note
While testing the encryption I ran into some unexpected problems:
- i used the wrong key-id (wrong format) for encryption, so decryption was not possible anymore. Instructions i found on the Internet were not very clear in this regard.
- i thought that the restore / decryption would not require a password (which would be a big no go). The problem turned out to be caching of the GPG agent, which had the file loaded after the first run.
In total, the above problems cost me about an hour. In the next part, we will look at offsite-backups (transferring the backup to another server).
Hope you learned some
Matthias
comments powered by Disqus