The novice said: "I will save my working files, but not my system and application files, as they can be always be reinstalled from their distribution disks."
The master made no reply.
The next day, the novice's disk crashed. Three days later, the novice was still reinstalling software.
If that's your situation, it might be worth backing up applications. However, there are better solutions that make this irrelevant. For instance, using NixOS, you can literally back up a single text file describing your system setup, and you can recreate the system from it automatically in minutes, guaranteed to be bit for bit identical.
In my experience that works in box like 80% of the time. Which is great! But a backup is great in case it doesn’t.
If you are wondering ‘How are you getting that much failure’, it’s because I like to freeze older versions and they often decay for one reason or another (can’t get files for example)
At the same time it is significantly more "hardware-change" resilient, since the same backed up config file (minus perhaps some very system specific lines that you may or may not have) can bring up a functionally equivalent working system on quite distinct other hardware as well.
If bare metal recovery takes shorter time than reinstalling, then you should make bare metal backups. When bare metal updates, so should your backups.
If you're using virtual machines, make sure to backup the hypervisor config and setup as well. You never know which part of the infrastructure starts to fail first.
If your NixOS setup is not of hello-world kind, you typically have sops in it (or another secret management tool) and want to backup your age keys as well. They're necessarily separated from the main config tree so they don't end up leaking to the globally accessible nix store, and in the most typical case they're located in a 600 file outside the git repo. Forgetting to back it up is a classic NixOS gotcha and also a generalized version of this koan.
I accidentally let qwen wipe my 60tb nas the other day, and the tao of that is "well at least i dont have to stress about what im doing with my 30 years of rando .docs, facebook pics, games, ebooks and star trek episodes anymore"
c'est la vie
Like giving a 6yo unix jr eng a crayon and root login
> The master said, "Do not despair, for yesterday I took one of your backup tapes and posted it to my brother in China. He will return it."
The master should not have done this, even by sea mail, for the very first law of backup is: you will only learn to make proper backups after you suffer a catastrophic loss of data.
Protecting the student from this is depriving him of the only true knowledge, the one that is not merely stored in the head, but burned into the skin.
He loaded the week-old backup, but it was the same. Eventually, he realized that the virus had struck four months before. He returned to the master and said: "Master, you have the only uncorrupted copy!"
But the master had already overwritten it with a copy of DOOM.
> Store your Really Important Info in a Google Drive/iCloud synced VeraCrypt volume
There is no way I would entrust Google or Apple with keeping important data. Google being infamous for account issues with absolutely no support and Apple already borked my account in a way that makes my iPad unusable the moment I turn on backup sync.
Storage is Hetzner Storage Box which so far feels great. Cheap and supports SSH SFTP WebDAV, so you can use rsync or restic and browse artifacts over www.
It feels a little slow hosting in Germany but using from US. But I haven’t found a provider with the same cost / features.
NetDynamics shares some promo codes every so often, they have a data center in Utah as well as the EU. I wouldn't call it super low latency -- but backup generally doesn't need to be. I've used both at various times for a personal off site backup target.
But for commercial use, IDK, you'd want to dig into the details of their system. Not exactly S3 levels of resilience, but perhaps enough for some people, I leave it to others to judge.
After six years on my old build, last week I built a new computer. One of the things I invested in was treating the backup regime as the most important aspect of the new build. There's a piece of mind one gets from achieving the "seven heads"
A novice asked the master: “What must I do to learn the Zen of Backup?” The master whacked the novice in the head, grabbed the keyboard, and ran sudo rm -rf /. The novice smiled and left feeling enlightened.
I worked for a small manufacturing company some years back, and depending on what you're doing it's easy to rack up some huge file storage usage and big databases without being in the business of video, photography or music. The database that only ran our manufacturing, not inclusive of HR, accounting, etc., was about 1TB. The design files that went to the plant (most of which were just text, you could open them with Notepad) were another 3 or 4 TB worth. Maybe another 1TB or so for various collateral bits of data related to jobs, mostly PDFs.
The company has, I'm sure, probably tripled all that by now. In their case it compressed really well, but still.
The $/TiB storage cost currently creates weird choices in backup media... even making some older media formats economical again. Just finished a consumer storage option survey yesterday.
Even if you account for consumer drive+spare cost, a 8TB HDD is still the best JBOD option. For under several dozen 25GB projects, the BD-R disc drive is currently the cheapest media backup option (about 35% cheaper than HDD above 12TiB). Note 50G and 100G BD EOL media are actually equal to or a worse deal than HDD.
Also, HPE LTO9 18GB Tapes are currently around 5 to 15 times cheaper than HDD and SSD in terms of data storage cost, but the fixed cost of tape auto-loaders and drives don't always make sense until you have a significant amount of backup data. Tapes may also create a single point of failure in use... so incremental backups are a non-zero risk, and triple redundant hardware is required (old hardware gets hard to find.)
The classic joke is still true: "Fast, cheap, or good.. choose any two..."
I looked in to the bluray thing and concluded it just doesn't make sense at any level. You have to buy a bluray writer which will become increasingly rare, and then you are limited to 25GB disks where you have to factor in a chunk of that will go unused as each project won't fit on exactly 25gb.
For the effort you are expending, just getting a hard drive makes way more sense even if the $/gb is technically a little worse. The utility of hard drives is much better.
Blu-ray, or any non-rewritable optical media, guarantees that it hasn't been tampered with. If I were attacked by malware, I like the confidence that the Blu-ray backup is absolutely guaranteed to be the same as on the date it was written. A hard disk used for backup might have been reattached to the network at some point after the initial backup and gotten infected/corrupted/altered.
> Blu-ray, or any non-rewritable optical media, guarantees that it hasn't been tampered with.
In terms of external tampering, I think that need is better-served by encrypting everything, which you probably want to do anyway. Or at least using some sort of checksum signed by a key.
However if the focus is "what if the backup process is compromised", then yes, having some date which is beyond its writing-reach may be useful. Another aspect of this might be a remote-backup system where one set of credential is enough for daily backups and restore, while another is needed in order to access a set of older duplicates that are remotely maintained as ransomware-protection.
That depends on your threat model. If the attacker can swap the Blu-ray disc to a similar looking one, but with content they control, that absolute guarantee becomes moot.
>I looked in to the bluray thing and concluded it just doesn't make sense at any level.
I mostly agree, but the 25GB M-Disc EOL media format is fairly interesting for archival purposes. The gamble actually made sense in this rare instance as each active project folder still fit on one disc, and the $120USD omnidrive firmware drives should keep the retro-equipment running for a few more years. These are relatively inexpensive even with a spare unit in the box.
>just getting a hard drive makes way more sense
In many cases this can be true, but the 3 years of HDD flight time compared to the 7+ year SSD wear life is also fairly close. It is weird, but the cost of higher capacity 18TB/24TB HDD also no longer make sense for JBOD either... unless you are hitting rack space constraints. =3
Just bought an LTO drive yesterday (after failing to cool its predecessor properly - dont be like me). They got more expensive more or less in line with HDDs :(
Used Tape drives often had the write heads service hours go well past its recommended limits. Mission critical IT systems don't get "upgraded" off schedule very often. ymmv
Anecdotally, I have seen Tape sets fail in production settings, as a remote office location admin was doing incremental backups... and the sick tape drive kept silently damaging each daily roll-back attempt. They may fail ungracefully in an archival context.
I do not deny a janky BD-R drive option is situation specific, but for small projects I am fine with a 25GB disc $25.80/TiB solution with the occasional big file $62.42/TiB hit for 100G/128G EOL discs.
HDD are less hassle, still hit below $34.63/TiB, and last a few years. Not a bad fall back option if you still have old inventory. =3
Depending on the amount of data you have, might it not be less expensive than build the tape drive?
An LTO drive is easily $10.000. You can restore a lot of S3 data for that amount of money. Sadly getting cheaper tape options is not really possible anymore.
It isn't too expensive for my use case – per month, I retrieve about 500GB a month from Glacier Instant Retrieval to verify, which costs ~$15. Combined with other costs, my monthly bill is ~$50.
I used to use Backblaze B2, which had cheaper egress at the cost of slightly more expensive storage (I think ~$5/TB/mo, with free egress up to 3x all current storage?) So I paid $20. But at some point AWS opened a datacenter near me, and I couldn't ignore how much faster data transfers got.
AWS is an option up to 30TiB, but also feeds into the over-provisioned data-center market pressures. The recent "rclone --gui" solution does bestow rsync/sshfs like features for Windows users now, and remote mounts are easier/cleaner to bring up. =3
I spent lots of time thinking about backups. I've got files dating back to 1991 and even some from the 80s (but I don't care about those so didn't bother backing them up: when the last 5"1/4 shall stop reading, those will be gone and it's fine).
I've even got stuff like copies of early websites made by friends, websites long gone and I'm pretty sure I'm the only one to still have backups of those.
To me verifying the backup should be part of the backup procedure itself and so I did just that: my backup procedure does verify that the backups can be decrypted/unarchived and it then verifies the files inside the backup.
Now the thing is: you don't need to verify 100% of your backups all the time (as in I don't decrypt and verify the checksums of 100% of my backups all the time). You can do random sampling: over n backups, you can be reasonably sure at least x% of the files are correct.
So random sampling is part of my backup procedure.
As those are encrypted backups, the verification also ensures that the backups can be decrypted (would be too bad otherwise, wouldn't it?).
I also believe the backuping procedure should not be able to go wild and destroy data. So my backup procedure is done by a podman container that access the volume with all my data (the one that needs to be backed up) read-only.
> To believe in one's backups is one thing. To have to use them is another."
Yes, which is why a proper backup procedure verifies the backup it just created. The greenlight is only given to the backup that's just been made once it's been verified by my verification script.
Some stuff can be verified automatically: for example say you backup a Git repo: a strict, full, git fsck on the backup (that is: on the Git repo once pulled out of the backup, before greenlighting it) works fine.
For other things I've got my own verifications.
> These hundreds of corrupted files have been flowing through my backup system. Now I do not know which files are clean and which are not.
Which is why many of my files, which I know aren't supposed to ever change anymore, have a partial checksum added as part of the filename.
I don't have:
DSC0983747.JPG
but:
DSC0983747-b3-7e228491a0.JPG
where 7e228491a0 are the first 40 bits of the Blake3 checksum of the file.
Then it become very complicated for "something" (bit rot or malicious) to silently corrupt my stuff for the backuping procedure (which only has read-only access to data, remember) goes crazy bonkers and warns me as soon as two identically named files (one on the last backup and the current version) have an identical name but different content and one doesn't match the checksum (this is cause for a "stop the world" and immediate enquiry: and, yup, it already allowed me to catch a corrupted file).
I've moreover got a sheet of paper, laminated, that explains how to access the backups and that explanation is saved in many places (safe at the bank, safe at my brother's house in another country, etc.): should say my place burn with me inside, but my wife be safe, she'd be able to access the backups too.
As for my main data, it's RAID on ZFS on a machine with ECC and that's where the backups are made (and automated) from.
Proper backup procedure is not hard: it just takes some time to set up properly, once. Then it's done for a lifetime.
If you are wondering ‘How are you getting that much failure’, it’s because I like to freeze older versions and they often decay for one reason or another (can’t get files for example)
If you're using virtual machines, make sure to backup the hypervisor config and setup as well. You never know which part of the infrastructure starts to fail first.
> The master said: "Now you are truly enlightened."
c'est la vie
Like giving a 6yo unix jr eng a crayon and root login
The master should not have done this, even by sea mail, for the very first law of backup is: you will only learn to make proper backups after you suffer a catastrophic loss of data.
Protecting the student from this is depriving him of the only true knowledge, the one that is not merely stored in the head, but burned into the skin.
There is no way I would entrust Google or Apple with keeping important data. Google being infamous for account issues with absolutely no support and Apple already borked my account in a way that makes my iPad unusable the moment I turn on backup sync.
* windows * android
* specific applications like
* signal * photos
and son on :D
It's how I learned to back up data.
Well. That, and data loss.
Perusing small blogs over feeds any day of the week.
Storage is Hetzner Storage Box which so far feels great. Cheap and supports SSH SFTP WebDAV, so you can use rsync or restic and browse artifacts over www.
It feels a little slow hosting in Germany but using from US. But I haven’t found a provider with the same cost / features.
Any alternatives folks use for backups?
https://deadca7.com/blog/backups
But for commercial use, IDK, you'd want to dig into the details of their system. Not exactly S3 levels of resilience, but perhaps enough for some people, I leave it to others to judge.
A novice asked the master: “What must I do to learn the Zen of Backup?” The master whacked the novice in the head, grabbed the keyboard, and ran sudo rm -rf /. The novice smiled and left feeling enlightened.
Other than photos, music and videos, everything I have ever done in the last 40 years is 1.57Gb.
Media is where it got expensive.
The company has, I'm sure, probably tripled all that by now. In their case it compressed really well, but still.
All that to say, YMMV, significantly.
Even if you account for consumer drive+spare cost, a 8TB HDD is still the best JBOD option. For under several dozen 25GB projects, the BD-R disc drive is currently the cheapest media backup option (about 35% cheaper than HDD above 12TiB). Note 50G and 100G BD EOL media are actually equal to or a worse deal than HDD.
Also, HPE LTO9 18GB Tapes are currently around 5 to 15 times cheaper than HDD and SSD in terms of data storage cost, but the fixed cost of tape auto-loaders and drives don't always make sense until you have a significant amount of backup data. Tapes may also create a single point of failure in use... so incremental backups are a non-zero risk, and triple redundant hardware is required (old hardware gets hard to find.)
The classic joke is still true: "Fast, cheap, or good.. choose any two..."
Best of luck =3
For the effort you are expending, just getting a hard drive makes way more sense even if the $/gb is technically a little worse. The utility of hard drives is much better.
In terms of external tampering, I think that need is better-served by encrypting everything, which you probably want to do anyway. Or at least using some sort of checksum signed by a key.
However if the focus is "what if the backup process is compromised", then yes, having some date which is beyond its writing-reach may be useful. Another aspect of this might be a remote-backup system where one set of credential is enough for daily backups and restore, while another is needed in order to access a set of older duplicates that are remotely maintained as ransomware-protection.
I mostly agree, but the 25GB M-Disc EOL media format is fairly interesting for archival purposes. The gamble actually made sense in this rare instance as each active project folder still fit on one disc, and the $120USD omnidrive firmware drives should keep the retro-equipment running for a few more years. These are relatively inexpensive even with a spare unit in the box.
>just getting a hard drive makes way more sense
In many cases this can be true, but the 3 years of HDD flight time compared to the 7+ year SSD wear life is also fairly close. It is weird, but the cost of higher capacity 18TB/24TB HDD also no longer make sense for JBOD either... unless you are hitting rack space constraints. =3
Anecdotally, I have seen Tape sets fail in production settings, as a remote office location admin was doing incremental backups... and the sick tape drive kept silently damaging each daily roll-back attempt. They may fail ungracefully in an archival context.
I do not deny a janky BD-R drive option is situation specific, but for small projects I am fine with a 25GB disc $25.80/TiB solution with the occasional big file $62.42/TiB hit for 100G/128G EOL discs.
HDD are less hassle, still hit below $34.63/TiB, and last a few years. Not a bad fall back option if you still have old inventory. =3
An LTO drive is easily $10.000. You can restore a lot of S3 data for that amount of money. Sadly getting cheaper tape options is not really possible anymore.
Ah that sucks :(
Rule #25: Try to avoid making what one may purchase, as the total cost is often many times higher than first assumed.
It is tempting for many, but often doesn't end well for businesses. =3
I used to use Backblaze B2, which had cheaper egress at the cost of slightly more expensive storage (I think ~$5/TB/mo, with free egress up to 3x all current storage?) So I paid $20. But at some point AWS opened a datacenter near me, and I couldn't ignore how much faster data transfers got.
The point is the cost doesn't scale well.
A master after my own heart.
I've even got stuff like copies of early websites made by friends, websites long gone and I'm pretty sure I'm the only one to still have backups of those.
To me verifying the backup should be part of the backup procedure itself and so I did just that: my backup procedure does verify that the backups can be decrypted/unarchived and it then verifies the files inside the backup.
Now the thing is: you don't need to verify 100% of your backups all the time (as in I don't decrypt and verify the checksums of 100% of my backups all the time). You can do random sampling: over n backups, you can be reasonably sure at least x% of the files are correct.
So random sampling is part of my backup procedure.
As those are encrypted backups, the verification also ensures that the backups can be decrypted (would be too bad otherwise, wouldn't it?).
I also believe the backuping procedure should not be able to go wild and destroy data. So my backup procedure is done by a podman container that access the volume with all my data (the one that needs to be backed up) read-only.
> To believe in one's backups is one thing. To have to use them is another."
Yes, which is why a proper backup procedure verifies the backup it just created. The greenlight is only given to the backup that's just been made once it's been verified by my verification script.
Some stuff can be verified automatically: for example say you backup a Git repo: a strict, full, git fsck on the backup (that is: on the Git repo once pulled out of the backup, before greenlighting it) works fine.
For other things I've got my own verifications.
> These hundreds of corrupted files have been flowing through my backup system. Now I do not know which files are clean and which are not.
Which is why many of my files, which I know aren't supposed to ever change anymore, have a partial checksum added as part of the filename.
I don't have:
but: where 7e228491a0 are the first 40 bits of the Blake3 checksum of the file.Then it become very complicated for "something" (bit rot or malicious) to silently corrupt my stuff for the backuping procedure (which only has read-only access to data, remember) goes crazy bonkers and warns me as soon as two identically named files (one on the last backup and the current version) have an identical name but different content and one doesn't match the checksum (this is cause for a "stop the world" and immediate enquiry: and, yup, it already allowed me to catch a corrupted file).
I've moreover got a sheet of paper, laminated, that explains how to access the backups and that explanation is saved in many places (safe at the bank, safe at my brother's house in another country, etc.): should say my place burn with me inside, but my wife be safe, she'd be able to access the backups too.
As for my main data, it's RAID on ZFS on a machine with ECC and that's where the backups are made (and automated) from.
Proper backup procedure is not hard: it just takes some time to set up properly, once. Then it's done for a lifetime.