Network/Backup: Difference between revisions

From NURDspace
(Created page with "= Host backups = On erratic, we have a 2 x 4TB raid1 array on which we have installed a ZFS filesystem. This filesystem is mounted on backup.vm.nurd.space, and using a bunch o...")
 
 
(8 intermediate revisions by 2 users not shown)
Line 1: Line 1:
= Host backups =
= Host backups =
On erratic, we have a 2 x 4TB raid1 array on which we have installed a ZFS filesystem. This filesystem is mounted on backup.vm.nurd.space, and using a bunch of scripts, we manage backups. The architecture we use is a file-based pull-based one using rsync. On the backup server is a config file in which all hosts that are to be backed up are placed. Each day, cron will trigger running of a script, which will read in the config file, and make full backups of configured hosts. Cron will also trigger a script which will make a daily snapshot of the zfs filesystem for a month. For snapshots older then a month, all but the first of the month will be pruned.
On erratic, we have a 2 x 4TB raid1 array on which are passthrough to a VM running Proxmox Backup Server. All the backups of VMs are handled by Proxmox. Every day at 8am a backup job will run.  
To get your VM or lxc backed up, you do not have to do anything. It will be automatically included. If you do not want this to happen, you can exclude your VM/ lxc in datacenter > backup > edit backup job


= Adding/removing hosts from backup =
= Adding/removing hosts from backup =
To add or remove a host from backup, modify the file /etc/backup.hosts. This file contains a comma-separated list of hosts to backup. The first column is the fqdn of the host which needs to be backed up. The second is the distro for this host. Right now, debian and openbsd are supported as hosts. Note that the host that is to be backed up needs to have a public key installed into the root account. This is included if you deploy the host as a [[Network/ManagedVM|ManagedVM]].
To add or remove a host from backup, modify the file /etc/backup.hosts. This file contains a comma-separated list of hosts to backup. The first column is the fqdn of the host which needs to be backed up. The second is the distro for this host. Right now, debian and openbsd are supported as hosts. Note that the host that is to be backed up needs to have a public key installed into the root account. This is included if you deploy the host as a [[Network/ManagedVM|ManagedVM]].


= Schedules =
== Proxmox Backup Server (PBS) ==
* /usr/local/sbin/run_backups.sh runs every day at 0500
For more information about this, visit the [https://pbs.proxmox.com/docs/index.html docs].  
* /usr/local/sbin/manage_snapshots.sh runs every day at 2355


= Restores =
PBS can be reached from within the space network at https://backup.vm.nurd.space:8007
If you need to restore a file, login to the backup server, and navigate to the /backup folder. In here, you will find a subdir for each host that is in the backup. Navigate to the subdir of this host, and find the file(s) you are looking for. If you need to restore a file from a snapshot, navigate to the /backup/.zfs/snapshot directory, select the snapshot you want, and navigate to the host directory underneath this snapshot dir.


= run_backups.sh =
For some additional details see [[Network/Roadmap/RebuildBackupServer]]
#!/bin/bash
== Schedules ==
* Every day at 8:00, Erratic will run the backup job for all the VMs and LXCs
PATH='/sbin:/usr/sbin:/usr/local/sbin:/bin:/usr/bin:/usr/local/bin'
* Every day at 0:00, a prune job will run that removes old backups (daily 7, monthly 1, yearly 1)
export PATH
* Every month at 0:00, a verify job will run.
BACKUP_HOSTS="/etc/backup.hosts"
KNOWN_HOSTS="/root/.ssh/known_hosts"
BACKUP_SSH_KEY="/root/.ssh/id_ed25519_backup"
function log {
  logger -t $(basename ${0}) ${@}
  echo "[$(date)] ${@}"
}
function do_backup {
  HOST="${1}"
  PLATFORM="${2}"
  grep -q "${HOST}" ${KNOWN_HOSTS}
  if [[ ${?} -eq 1 ]]; then
    log "fetching ssh host keys for ${HOST}"
    ssh-keyscan ${HOST} 2>/dev/null >> ${KNOWN_HOSTS}
  fi
  log "running backup for ${HOST}"
  # We have a couple of machines that have /home mounted from harmony
  # While it is okay to have one of these in the backup, we dont need
  # multiple copies, so create an exclude for all but the first user
  # of this share (egg.vm.nurd.space)
  EXCLUDE_HOME=""
  if [[ ${HOST} == "nurdservices.lan.nurd.space" ]]; then
    EXCLUDE_HOME="--exclude=/home"
  fi
  case "${PLATFORM}" in
    "debian")
      rsync -avpl \
        -e "ssh -i ${BACKUP_SSH_KEY}" \
        --exclude='/backup' \
        --exclude='/music' \
        --exclude='/mnt/mp3' \
        --exclude='/mnt/pve' \
        --exclude='/dev' \
        --exclude='/proc' \
        --exclude='/run' \
        --exclude='/sys' \
        --exclude='/tmp' \
        --exclude='/lost+found' \
        --exclude='/var/cache' \
        --exclude='/var/crash' \
        --exclude='/var/lock' \
        --exclude='/var/run' \
        --exclude='/var/spool' \
        --exclude='/var/tmp' \
        --exclude='/var/lib/containers' \
        --exclude='/var/lib/podman' \
        --exclude='/var/lib/libvirt/images' \
        --exclude='/var/lib/kubelet/pods' \
        --exclude='/var/lib/docker/containers' \
        --exclude='/var/lib/docker' \
        --exclude='/var/lib/rancher/k3s/agent/containerd' \
        ${EXCLUDE_HOME} \
        --delete-after \
        ${HOST}:/ /backup/${HOST}/
      ;;
    "openbsd")
      rsync -avpl \
        -e "ssh -i ${BACKUP_SSH_KEY}" \
        --exclude='/dev' \
        --exclude='/tmp' \
        --exclude='/var/cache' \
        --exclude='/var/run' \
        --exclude='/var/spool' \
        --exclude='/var/sysmerge' \
        --exclude='/var/syspatch' \
        --exclude='/var/tmp' \
        --delete-after \
        ${HOST}:/ /backup/${HOST}/
      ;;
    *)
      rsync -avpl \
-e "ssh -i ${BACKUP_SSH_KEY}" \
        --exclude='/proc' \
--exclude='/sys' \
--exclude='/dev' \
--exclude='/run' \
--exclude='/var/run' \
--exclude='/tmp' \
--exclude='/var/tmp' \
--delete-after \
${HOST}:/ /backup/${HOST}/
      ;;
  esac
}
# Create backup of hosts on this site
cat ${BACKUP_HOSTS} | while read DATA; do
  HOST="$(echo ${DATA} | cut -d, -f1)"
  PLATFORM="$(echo ${DATA} | cut -d, -f2)"
  do_backup "${HOST}" "${PLATFORM}"
done


= manage_snapshots.sh =
== Remote Hosts ==
#!/bin/bash
PBS supports remote hosts as well, and our remote hosts are configured to backed up to PBS (soon).
 
PATH='/sbin:/usr/sbin:/usr/local/sbin:/bin:/usr/bin:/usr/local/bin'
Ask a friendly BOFH to give you a username and password, as well as make a namespace. (ideally we set this up to use ldap instead)
  export PATH
 
   
 
  TODAY="$(date +%d-%m-%Y)"
=== Setting up on a Debian bookworm host ===
VOLUMES="$(zfs list | awk '/zbackup\//{print $1}')"
'''Installing proxmox backup client'''
sudo wget https://enterprise.proxmox.com/debian/proxmox-release-bookworm.gpg -O
  function log {
/etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg
  logger -t $(basename ${0}) ${@}
echo "deb http://download.proxmox.com/debian/pbs-client bookworm main" >>
  echo "[$(date)] ${@}"
  /etc/apt/sources.list.d/pbs-client.list
  }
  apt update
   
  sudo apt install proxmox-backup-client
# Create snapshot for today
 
for VOLUME in ${VOLUMES}; do
'''Making a backup'''
  SNAPSHOT_NAME="${VOLUME}@${TODAY}"
 
  zfs list -t snapshot 2>/dev/null | grep -q "${SNAPSHOT_NAME}"
  export PBS_REPOSITORY="<your username>@pbs@backup.vm.nurd.space:backup"
  if [[ ${?} -eq 1 ]]; then
  export PBS_FINGERPRINT="f4:f7:d8:01:13:5e:bd:0b:a6:b1:31:89:39:b6:4c:fa:03:69:82:17:cf:45:96:2c:bd:a9:08:bd:7f:ee:50:9d"
    log "creating snapshot ${SNAPSHOT_NAME}"
  export PBS_PASSWORD="<your password>"
    zfs snapshot "${SNAPSHOT_NAME}"
proxmox-backup-client backup <name>.pxar:/<path>--ns <your namespace>
  fi
 
  done
Another example, taken from Proxmox
  /proxmox-backup-client backup --crypt-mode=none root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --ns erratic
# Cleanup last months snapshots
 
DAYNUM=$(date +%d)
For more information see https://pbs.proxmox.com/docs/backup-client.html
if [[ ${DAYNUM} -eq 1 ]]; then
  MONTHNUM=$(date +%m)
  YEAR=$(date +%Y)
  LASTMONTH=$(expr ${MONTHNUM} - 1)
  if [[ ${MONTHNUM} -eq 1 ]]; then
    LASTMONTH=12
  fi
  FILTER="$(printf "%02d-${YEAR}" ${LASTMONTH})"
  SNAPSHOTS=$(zfs list -t snapshot | awk "/${FILTER}/{print \$1}" | grep -v "01-${FILTER}")
  for SNAPSHOT_NAME in ${SNAPSHOTS}; do
    log "removing snapshot ${SNAPSHOT_NAME}"
    zfs destroy "${SNAPSHOT_NAME}"
  done
fi

Latest revision as of 16:21, 2 August 2023

Host backups

On erratic, we have a 2 x 4TB raid1 array on which are passthrough to a VM running Proxmox Backup Server. All the backups of VMs are handled by Proxmox. Every day at 8am a backup job will run. To get your VM or lxc backed up, you do not have to do anything. It will be automatically included. If you do not want this to happen, you can exclude your VM/ lxc in datacenter > backup > edit backup job

Adding/removing hosts from backup

To add or remove a host from backup, modify the file /etc/backup.hosts. This file contains a comma-separated list of hosts to backup. The first column is the fqdn of the host which needs to be backed up. The second is the distro for this host. Right now, debian and openbsd are supported as hosts. Note that the host that is to be backed up needs to have a public key installed into the root account. This is included if you deploy the host as a ManagedVM.

Proxmox Backup Server (PBS)

For more information about this, visit the docs.

PBS can be reached from within the space network at https://backup.vm.nurd.space:8007

For some additional details see Network/Roadmap/RebuildBackupServer

Schedules

  • Every day at 8:00, Erratic will run the backup job for all the VMs and LXCs
  • Every day at 0:00, a prune job will run that removes old backups (daily 7, monthly 1, yearly 1)
  • Every month at 0:00, a verify job will run.

Remote Hosts

PBS supports remote hosts as well, and our remote hosts are configured to backed up to PBS (soon).

Ask a friendly BOFH to give you a username and password, as well as make a namespace. (ideally we set this up to use ldap instead)


Setting up on a Debian bookworm host

Installing proxmox backup client 

sudo wget https://enterprise.proxmox.com/debian/proxmox-release-bookworm.gpg -O 
/etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg
echo "deb http://download.proxmox.com/debian/pbs-client bookworm main" >> 
/etc/apt/sources.list.d/pbs-client.list
apt update
sudo apt install proxmox-backup-client

Making a backup 

export PBS_REPOSITORY="<your username>@pbs@backup.vm.nurd.space:backup"
export PBS_FINGERPRINT="f4:f7:d8:01:13:5e:bd:0b:a6:b1:31:89:39:b6:4c:fa:03:69:82:17:cf:45:96:2c:bd:a9:08:bd:7f:ee:50:9d"
export PBS_PASSWORD="<your password>"
proxmox-backup-client backup <name>.pxar:/<path>--ns <your namespace>

Another example, taken from Proxmox 

/proxmox-backup-client backup --crypt-mode=none root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --ns erratic

For more information see https://pbs.proxmox.com/docs/backup-client.html

Retrieved from "https://nurdspace.nl/index.php?title=Network/Backup&oldid=13592"
Last modified
This page was last edited on 2 August 2023, at 16:21.