projects/infrastructure/server-backup-with-rsync.md

# Server Backup With Rsync

First of all make sure that rsync is installed on the server: `apt install rsync`


## 1. Create a backup user with ssh-key access

1. Create a backup user, for example `backup1` (in ubuntu there is already a user named `backup`):

   ```
   useradd backup1 -m
   ls -al /home/backup1/
   ```

1. Create a ssh key-pair for this user:

   ```
   ssh-keygen -t ecdsa -P '' -q -f key1

   ls -l key1*
   cat key1
   cat key1.pub
   ```

1. Add the public key to `/home/backup1/.ssh/authorized_keys`:

   ```
   mkdir -p /home/backup1/.ssh
   chown backup1: /home/backup1/.ssh
   chmod 700 /home/backup1/.ssh

   cat key1.pub >> /home/backup1/.ssh/authorized_keys
   chown backup1: /home/backup1/.ssh/authorized_keys
   chmod 600 /home/backup1/.ssh/authorized_keys

   ls -al /home/backup1/.ssh/
   cat /home/backup1/.ssh/authorized_keys
   ```

1. Try to login with this key:

   ```
   ssh -p 22 -i key1 backup1@localhost
   ```

   You should be able to login without a password.

1. Try to copy something:

   ```
   mkdir -p /home/backup1/test1
   touch /home/backup1/test1/file1.txt
   touch /home/backup1/test1/file2.txt
   ls -al /home/backup1/test1

   rsync -a -e "ssh -p 22 -i key1" backup1@localhost:~/test1 .

   ls -al test1
   ```

## 2. Restrict the ssh key of the backup user for using only rsync

1. Let's find out the command that the client is sending to the server through SSH.
   Let's try the same rsync command again, with the added SSH switch `-v` (verbose):

   ```
   rsync -a -e "ssh -p 22 -i key1 -v" backup1@localhost:~/test1 .
   ```

   Then let's look for the debug line that says "Sending command":

   ```
   rsync -a -e "ssh -p 22 -i key1 -v" backup1@localhost:~/test1 . 2>&1 | grep "Sending command"
   ```

   It should be something like this:

   ```
   rsync --server --sender -logDtpre.iLsfxC . ~/test1
   ```

1. We can restrict the SSH key `key1` to execute only this command and nothing else.
   For this we need to add something like this before the public key on `/home/backup1/.ssh/authorized_keys`:

   ```
   command="rsync --server --sender -logDtpre.iLsfxC . ~/test1" ecdsa-sha2-nistp256 AAAAE2Vj....
   ```

   To make it even more secure, we can also add the options `no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding`. The file `/home/backup1/.ssh/authorized_keys` now should look like this:

   ```
   command="rsync --server --sender -logDtpre.iLsfxC . ~/test1",no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMbMdR9uW4SMeinpVvr6UQZaFybkiVZxm2DRYxFlCuxHchpTMGR7U4gZGZwY4D5LQDDy1Py4TWSsEizda4LecgQ= root@server
   ```

1. Let's check that now we cannot login with `key1` anymore, but we can still use it to `rsync`:

   ```
   ssh -p 22 -i key1 backup1@localhost  # should fail

   rm -rf test1
   rsync -a -e "ssh -p 22 -i key1" backup1@localhost:~/test1 .
   ls -l test1

   rm -rf test1
   rsync -a -e "ssh -p 22 -i key1" backup1@localhost: .
   ls -l test1
   ```

## 3. Create a read-only view of the parts of the filesystem that need to be backed up

1. Install `bindfs`:

   ```
   apt list bindfs
   apt show bindfs
   apt install bindfs
   ```

2. Create mount directories:

   ```
   mkdir -p /mnt/backup-server/scripts
   mkdir -p /mnt/backup-server/apps
   ```

3. Add these lines to `/etc/fstab` for mounting directories read-only:

   ```console
   /opt/docker-scripts /mnt/backup-server/scripts fuse.bindfs perms=0000:u=rD,force-user=backup1,force-group=nogroup 0 0
   /var/ds             /mnt/backup-server/apps    fuse.bindfs perms=0000:u=rD,force-user=backup1,force-group=nogroup 0 0
   ```

   Since we are using **docker-scripts** for installing and managing apps, these two directories
   are what we need to backup: `/opt/docker-scripts` and `/var/ds`.

4. Mount them:

   ```console
   mount -a
   ls -al /mnt/backup-server/scripts
   ls -al /mnt/backup-server/apps
   ```

5. Test that they are read-only:

   ```console
   sudo -u backup1 ls -al /mnt/backup-server/scripts
   sudo -u backup1 touch /mnt/backup-server/scripts/test1.txt
   ```

## 4. Create and use a backup script

1. For convenience, we can combine the command and the key in a bash script named `backup-server.sh`
   that looks like this:

   ```
   #!/bin/bash

   server=127.0.0.1
   port=22
   
   cd $(dirname $0)
   rsync -a -e "ssh -p $port -i $0" backup1@${server}: .
   
   exit 0
   
   -----BEGIN EC PRIVATE KEY-----
   MHcCAQEEIGHMv+mlPZg/V6TmLADsfZcpMRcfxykJGzXAF65tuQ4AoAoGCCqGSM49
   AwEHoUQDQgAExsx1H25bhIx6KelW+vpRBloXJuSJVnGbYNFjEWUK7EdyGlMwZHtT
   iBkZnBjgPktAMPLU/LhNZKwSLN1rgt5yBA==
   -----END EC PRIVATE KEY-----
   ```

   Let's try it:

   ```
   chmod 700 backup-server.sh
   rm -rf test1/
   ./backup-server.sh
   ls -l test1/
   ```

1. Now we can move this script to the client (backup server), making sure to set
   the proper values for the variables `server` and `port`, and it should work.

1. Let's also fix the directory on the server that is being backed up. We should edit
   `/home/backup1/.ssh/authorized_keys` and change `~/test1` to `/mnt/backup-server`

1. On the client (computer that is receiving the backup), let's place the script `backup-server.sh`
   on a directory like `/var/backup`:

   ```
   mkdir -p /var/backup
   mv backup-server.sh /var/backup/
   
   cd /var/backup/
   ./backup-server.sh
   ```

1. Let's also create a cron job that runs this script periodically each week:

   ```
   cat <<EOF > /etc/cron.d/backup-server
   # backup the server each tuesday
   0 0 * * TUE  root  /var/backup/backup-server.sh
   EOF
   ```