TechDocs/TechnicalProcesses/Backup/Servers

Description

Since July 2019, we use borg to backup most of our servers. Before that, we used custom scripts running on the server "cleve". On this page, the setup is described, and you will find information on how to add a new server to the backup process, and restore data if necessary.

The principle is simple: Each servers cares for its own backups, and can only access its own backups. Every night, a cron job triggers a wrapper script (borg-wrapper) which runs borg to make a backup to a remote storage. The server identifies on the remote storage with an SSH key. With this key, the server can only modify its own backups.

A log file for each backup cron run is sent to the FSFE admin+backup@ address. It contains the --stats --verbose output of the borg create and prune commands, as well as the output of any pre- or post-backup commands, by default a list of all current borg archives.

We automatically run regular snapshots of the complete remote storage which cannot be modified by the individual servers. We also use a sub-account to restrict access to other directories on the remote storage. This way, we have some safety if the server is compromised.

Add new server

If you created a new server, please configure its backup immediately! In order to do so, clone the git repo fsfe-backup (access restricted to System Hackers). It contains an Ansible playbook which takes care of the following steps:

To add the new server, add the server's FQDN (e.g. server.fsfe.org) to the file inventory in the client section.

Afterwards, run ansible-vault create host_vars/server.fsfe.org to create the configuration for the new server. In the new file, insert the new password used to encrypt backups:

---
borgbackup_passphrase: aLongAndSecurePassword

Note that the password for these encrypted files is stored in vaultpw.pgp, a GPG encrypted file which only a few people have access to. Please only add new passwords (and servers) if you can open this file. In the new file, you can also overwrite default variables.

Now you can run the playbook: ansible-playbook -i inventory -l server.fsfe.org backup.yml. Note that if you run this for multiple servers at once, please add -f 1 as a parameter. Otherwise, rewriting the authorized_keys file on the remote storage might cause issues if accessed by multiple processes at once.

If everything went fine, please commit and push your changes to Git!

Run scripts before backup

It may be useful to run commands right before the backup, e.g. to dump databases.

By default, the wrapper script runs /root/bin/backup.sh before the backup. Add your commands in this script file and make it executable. For example, it could look like this:

echo "Dump Postgres databases"
sudo -u fsfe -- sh -c "cd / ; pg_dump > /home/fsfe/postgres_dump.sql"

The output will be mailed with the borg logs to the specified email address.

Tricks

Restore from backup

In order to restore content of a specific borg archive, you can mount it using the wrapper script:

  1. Create a mountpoint: mkdir /mnt/borg

  2. Find the archive you would like to mount: borg-wrapper list

  3. Mount the archive to the local mountpoint: borg-wrapper mount 20190716-1255 /mnt/borg

Now, you can search and copy the desired data. Afterwards, please unmount the mointpoint with umount /mnt/borg.

Extraction of a complete archive is not supported by the wrapper script as of today. But by inspecting its content in /usr/local/bin/borg-wrapper, it should be fairly easy to build the command for this.

Uninstall backup

To remove all files generated by the Ansible playbook, e.g. in order to disable backups of a client, follow these steps:

Troubleshooting

Mail from cron job not sent

This can happen both with exim4 and postfix:

TechDocs/TechnicalProcesses/Backup/Servers (last edited 2019-07-29 09:51:48 by max.mehl)