Using Hetzner Storageboxes as backup targets for Restic/Rclone

For my offsite backup strategy I’m relying on rclone and restic. Rclone is being used to encrypt/copy backups taken by Proxmox Backup Server (which takes backups of the VM’s/Linux Containers running on Proxmox VE, deduplicates and compresses the data) to remote storage, while restic is used to do deduplication/encryption and copying of data stored on my NAS to remote storage.

For a while I was using Backblaze B2 (object storage) as a backend, but when I started validating part of my backups using restic’s check functions, costs went up dramatically (B2 charges for storage, download and certain transactions).

After some searching I came across iDrive E2, an S3-compatible object storage provider. Pricing was better than B2 (no egress/download cost). Initial tests were a bit bumpy (unexpected downtimes, timeouts, …), but support was responsive and in the end worked quite well. Still not 100% sure about the service - the speed is all over the place depending on the time of day - there’s no consistency whatsoever, which makes me worry a bit about the scale of their backend.

I do keep some (too much?) history on my backups - I can go back years - and this history means that the size of the backups is significant. By the time I finished my testing on E2, I had already reached 2TiB - at which point the price was once again getting to a point that it was going to start to cost plenty.

Another round of investigations led me to the Hetzner Storage Box. A storage box reacheable over a lot of protocols - FTP(S), SFTP, SCP, Samba/CIFS, BorgBackup (with which I’ve dabbled in the past), Restic, Rclone, rsync over SSH, WebDAV, … Sizes go from 1TB up to 20TB, at a affordable flat-fee. The fee for 5TiB (€155/y) is lower than what I would be paying at E2 ($200/y), while giving me more flexibility (protocol support). I know I could’ve stayed at the 2TiB plan at E2, and pay the extra on top, but somehow I’m still a littlI’ve dealt with Hetzner in the past and never been dissapointed.

To minimize backup down-time, I spun up a CX11 Cloud server, installed rclone on it, and copied the data from iDrive E2 over to the StorageBox. Copying went fairly fast, and after a verification from my home network, I’ve now switched completely over to using Hetzner. To make it easier for myself, I created two sub-accounts on the storagebox, so that the two backup mechanisms I’m using are properly segregated.

My rclone config for Hetzner looks like

[hetzner_sub1]
type = sftp
host = uXXXXXX.your-storagebox.de
user = uXXXXXX-sub1
key_file = /home/backupuser/.ssh/key1
port = 23
md5sum_command = md5 -r
sha1sum_command = sha1 -r
shell_type = unix
idle_timeout = 0

[hetzner_sub1_encrypted]
type = crypt
password = <random pw>
remote = hetzner_sub1:foldername

[hetzner_sub2]
type = sftp
host = uXXXXXX.your-storagebox.de
user = uXXXXXX-sub2
key_file = /home/backupuser/.ssh/key2
port = 23
shell_type = unix
md5sum_command = md5 -r
sha1sum_command = sha1 -r
idle_timeout = 0

hetzner_sub1_encrypted is the target used by rclone directly, while hetzner sub2 is used by restic as a backend. Restic is doing the encryption there, so I don’t need an additional layer by rclone.

One thing I bumped into is that Hetzner only allows 10 connections (per sub-account) at the same time. If you go over this the connections get blocked - something that rclone nor restic like. After playing with the options with the amount of connections I’ve settled on --fast-list --transfers 3 --checkers 6 for rclone, and --fast-list --transfers 4 --checkers 4 for restic.