projects/infrastructure/server-backup-with-rsync.md

Server Backup With Rsync

First of all make sure that rsync is installed on the server: apt install rsync

1. Create a backup user with ssh-key access

1. Create a backup user, for example backup1 (in ubuntu there is already a user named backup):

   useradd backup1 -m
   ls -al /home/backup1/
   

2. Create a ssh key-pair for this user:

   ssh-keygen -t ecdsa -P '' -q -f key1
   
   ls -l key1*
   cat key1
   cat key1.pub
   

3. Add the public key to /home/backup1/.ssh/authorized_keys:

   mkdir -p /home/backup1/.ssh
   chown backup1: /home/backup1/.ssh
   chmod 700 /home/backup1/.ssh
   
   cat key1.pub >> /home/backup1/.ssh/authorized_keys
   chown backup1: /home/backup1/.ssh/authorized_keys
   chmod 600 /home/backup1/.ssh/authorized_keys
   
   ls -al /home/backup1/.ssh/
   cat /home/backup1/.ssh/authorized_keys
   

4. Try to login with this key:

   ssh -p 22 -i key1 backup1@localhost
   

   You should be able to login without a password.

5. Try to copy something:

   mkdir -p /home/backup1/test1
   touch /home/backup1/test1/file1.txt
   touch /home/backup1/test1/file2.txt
   ls -al /home/backup1/test1
   
   rsync -a -e "ssh -p 22 -i key1" backup1@localhost:~/test1 .
   
   ls -al test1
   

2. Restrict the ssh key of the backup user for using only rsync

1. Let’s find out the command that the client is sending to the server through SSH. Let’s try the same rsync command again, with the added SSH switch -v (verbose):

   rsync -a -e "ssh -p 22 -i key1 -v" backup1@localhost:~/test1 .
   

   Then let’s look for the debug line that says “Sending command”:

   rsync -a -e "ssh -p 22 -i key1 -v" backup1@localhost:~/test1 . 2>&1 | grep "Sending command"
   

   It should be something like this:

   rsync --server --sender -logDtpre.iLsfxC . ~/test1
   

2. We can restrict the SSH key key1 to execute only this command and nothing else. For this we need to add something like this before the public key on /home/backup1/.ssh/authorized_keys:

   command="rsync --server --sender -logDtpre.iLsfxC . ~/test1" ecdsa-sha2-nistp256 AAAAE2Vj....
   

   To make it even more secure, we can also add the options no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding. The file /home/backup1/.ssh/authorized_keys now should look like this:

   command="rsync --server --sender -logDtpre.iLsfxC . ~/test1",no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMbMdR9uW4SMeinpVvr6UQZaFybkiVZxm2DRYxFlCuxHchpTMGR7U4gZGZwY4D5LQDDy1Py4TWSsEizda4LecgQ= root@server
   

3. Let’s check that now we cannot login with key1 anymore, but we can still use it to rsync:

   ssh -p 22 -i key1 backup1@localhost  # should fail
   
   rm -rf test1
   rsync -a -e "ssh -p 22 -i key1" backup1@localhost:~/test1 .
   ls -l test1
   
   rm -rf test1
   rsync -a -e "ssh -p 22 -i key1" backup1@localhost: .
   ls -l test1
   

3. Create a read-only view of the parts of the filesystem that need to be backed up

1. Install bindfs:

   apt list bindfs
   apt show bindfs
   apt install bindfs
   

2. Create mount directories:

   mkdir -p /mnt/backup-server/scripts
   mkdir -p /mnt/backup-server/apps
   

3. Add these lines to /etc/fstab for mounting directories read-only:

   /opt/docker-scripts /mnt/backup-server/scripts fuse.bindfs perms=0000:u=rD,force-user=backup1,force-group=nogroup 0 0
   /var/ds             /mnt/backup-server/apps    fuse.bindfs perms=0000:u=rD,force-user=backup1,force-group=nogroup 0 0
   

   Since we are using docker-scripts for installing and managing apps, these two directories are what we need to backup: /opt/docker-scripts and /var/ds.

4. Mount them:

   mount -a
   ls -al /mnt/backup-server/scripts
   ls -al /mnt/backup-server/apps
   

5. Test that they are read-only:

   sudo -u backup1 ls -al /mnt/backup-server/scripts
   sudo -u backup1 touch /mnt/backup-server/scripts/test1.txt
   

4. Create and use a backup script

1. For convenience, we can combine the command and the key in a bash script named backup-server.sh that looks like this:

   #!/bin/bash
   
   server=127.0.0.1
   port=22
      
   cd $(dirname $0)
   rsync -a -e "ssh -p $port -i $0" backup1@${server}: .
      
   exit 0
      
   -----BEGIN EC PRIVATE KEY-----
   MHcCAQEEIGHMv+mlPZg/V6TmLADsfZcpMRcfxykJGzXAF65tuQ4AoAoGCCqGSM49
   AwEHoUQDQgAExsx1H25bhIx6KelW+vpRBloXJuSJVnGbYNFjEWUK7EdyGlMwZHtT
   iBkZnBjgPktAMPLU/LhNZKwSLN1rgt5yBA==
   -----END EC PRIVATE KEY-----
   

   Let’s try it:

   chmod 700 backup-server.sh
   rm -rf test1/
   ./backup-server.sh
   ls -l test1/
   

2. Now we can move this script to the client (backup server), making sure to set the proper values for the variables server and port, and it should work.

3. Let’s also fix the directory on the server that is being backed up. We should edit /home/backup1/.ssh/authorized_keys and change ~/test1 to /mnt/backup-server

4. On the client (computer that is receiving the backup), let’s place the script backup-server.sh on a directory like /var/backup:

   mkdir -p /var/backup
   mv backup-server.sh /var/backup/
      
   cd /var/backup/
   ./backup-server.sh
   

5. Let’s also create a cron job that runs this script periodically each week:

   cat <<EOF > /etc/cron.d/backup-server
   # backup the server each tuesday
   0 0 * * TUE  root  /var/backup/backup-server.sh
   EOF