Summary
Our current backup solution is running into limits. We need to rebuild this into something more usable. First decision will be to determine which plan we are going to use for this.
Plan 1
Rebuild current zfs pool into something more usable. Basically we need to move from a single zfs volume (/backup) for the whole backupset, to a volume per host (/backup/hostname). We also need to rework snapshots into per-host snapshots.
Plan 2
Scrap the current design and move to https://www.proxmox.com/en/proxmox-backup-server/overview, since this is also zfs based AND be integrated into our hypervisor. This would mean deleting backup.vm.nurd.space, and repurposing its disks with proxmox-backup-server.
Architecture
The solution should be lightweight to implement, and all managed systems should automatically become part of the backup process. There needs to be a dedicated server that we can use to dump backups on for devices that cannot use the solution (eg, network devices). System should support snapshots and dedup (if this is not to memory costly)