Tuesday, September 24, 2019

In defense of the bunch-of-disks backup for NAS servers


As storage gets cheaper, more and more people are using NAS devices.

Many of these are relatively small (6-10 Tb).  These can be backed up on a single external USB hard drive.

However, many exceed 20 Tb, and there are a lot of people out there running 40 Tb or more.

Of course, these people purchased their NAS specifically for the very high capacity, redundancy and fault-tolerance of these devices.  Many run two-fault-tolerant arrays, dual redundant power supplies, dual UPS, and at least one hot spare.  They are as bulletproof as possible.

However, they are not perfect.  So it only makes sense to try to have a last-resort backup of everything that's on a NAS.

So - how do we do that?


Options are limited:

1.  Go cloud storage.  Yeah, great if you want to spend $200+ per month.

Oh, and all those "hacks" for "free / unlimited" options - they're either gone, going, or never worked anyway.

2.  Buy a second NAS and mirror it.  Great if you want to spend $3,000 to $5,000 on a new NAS plus hard drives to fill it.

3.  Tape.  Great if you want to spend $2,000 on a used tape drive.  Paying $50/tape is not bad, if you can afford the drive.

4.  Buy a smaller NAS and backup only "critical" stuff.  Defeats the purpose.

5. Backup to a rented box in a data center.  I don't even know how much that costs - likely lots - but just try uploading 50 Tb over your connection.

That leaves just one option: back up to a bunch of hard drives.  It is quite possibly the cheapest and most robust solution.


Despite this, I see lots of people bash this solution mercilessly.  So I wanted to present my take on it.



Like anything, this solution obviously has disadvantages:

-  Yes, you will lose some data if a drive fails.  However, you will also lose data if a NAS, RAID array, USB drive or a tape fails, so that is hardly a flaw unique to this solution. 

There is no software that will let you easily copy vast amounts of data across multiple hard drives.  Such "spanning" is inherent in CD/DVD writing, and can be done for USB sticks, but doesn't work for hard drives.  This makes for a lot of somewhat tedious manual copying.

[Note:  Handy Backup claims this feature is "coming soon".]

-  Incremental backups are a practical impossibility.  So plan to spend some time annually - and over the course of some weeks - re-copying all of the data you copied last year.

-  It seems slow.  (Though I doubt it is much slower than other solutions.)


It also has a lot of advantages:

1.  Simple:  It's easy.  A little tedious, but any computer can copy files.  There is no need for special software or complicated data-processing techniques.

2.  Robust:  Assuming you avoid encryption and compression, if you lose one drive, all the rest of the data is OK.  There is no risk of breaking an archive or backup format by losing a single chunk.

Now, obviously, losing any of your last-resort backup in an actual disaster situation would be A Bad Thing.  But not as bad as losing 1 drive of a 10-drive set that does not function without that one drive.

Similarly, bad sectors or other similar faults on one drive - or even multiple drives - does not affect the larger backup set.  Yes, you will lose files, but you won't lose all the files in the set.

3.  Cheap:  This uses only hard drives plus some means to connect them (USB or SATA dock) - which many people who run NAS systems will have lying around anyway.  No expensive bits like a NAS or tape drive.

For a little extra boost, re-use all those older - but still working - hard drives that you retired from your NAS boxen.  Sure, you may end up with a stack of 20 drives - but it's cheap!

4.  Capacity:  Spinning rust drives have the highest data density of any data storage device on the market, storing more in less space and for less money than anything else.  Plus, the $/Gb gets cheaper by the month.

5.  Expandable:  Add more drives anytime.

6.  Flexible:  Use whatever drives suit - mix and match at will.  You can even use a mix of SATA, USB and NAS drives.  Can't do that with tape (or, at least, not as easily).

7.  Portable:  Drives can easily be moved offsite, to a safety deposit box, stored in a safe, or whatever.  Storing in ideal conditions is relatively easy.  Bringing the drives to the restore point - wherever it may be - is also very easy.

8.  Durable:  While it's not recommended you test this, modern hard drives can survive horrific drops and other abuse with zero damage.  Old-timers will faint from shock before the drives actually give up.

9.  Protectable:  Drives can be easily protected against drop, ESD, fire, water, etc.  A Nanuk 935 will hold ten 3.5" SATA HDDs, making them almost indestructible.   For extra protection, consider a fireproof safe, or just go offsite.

10.  Compatible:  You don't get any more standard than a SATA disk and a GPT boot table.   No matter what changes in technology we might have, these drives are going to be machine-compatible for years, if not decades, in the future.  (Try that with your Zip, Jumbo and  DAT drives.)

11.  Universal:  Tou can read these drives back on any machine, right now.  No need for a special tape drive, special software to decompress or de-archive, or anything.

OK, at worst, you'll need a USB dock or USB drive case that can be picked up at any store, anywhere, anytime.  With that, any computer - be it Windows, Mac or Linux - virtually anywhere in the world will be able to read and restore your backups right now.

12.  Reliable:  Some say that a hard drives will retain data only 1.5 years in storage - others say 5 years is fine, or maybe even 10-20 years, or possibly even longer.  This simply won't be any issue if you refresh data reasonably often - say, annually.

Other than that, spinning drives don't tend to break when they're not spinning, and these drives will spend 99% of their life in an inert state.  So the drives are likely to physically last an extremely long time.

13:  Redundancy:  Spread the risk by storing critical data multiple times on multiple different drives, or storing multiple backup sets in different locations.

A lot of "critical" data is actually really small - documents, PDFs, and the like.  Even photos don't take up that much space unless you're a photo hound.

So, it's usually easy to copy what you really can't live without onto every drive of a backup set, and you won't even lose hardly any capacity.  You net 5-10 copies for the price of 1, essentially.  (Try that with your backup software.)

14.  Isolation:  Offline cold-storage drives are air-gapped and are not susceptible to ransomware or other threats. 

Ransomware sucks, but it's not going away, and you can still get pwned even if you do absolutely nothing to prompt infection yourself.  So you'd better have cold backups somewhere.

Yes, some data will be out of date, and yes, you will lose some of it.  But would you rather lose some - or all?


This solution isn't for everyone, and it's obviously suited best for those that need a last-resort, just-in-case copy of vast amounts of relatively data.  Kind of a roll-your-own, personal Amazon Glacier.

But, for anyone needing to backup tens of terabytes reliably, without the need for instant restore, and without access to corporate-grade hardware, this looks like the best choice to me.


You should permanently install a USB boot / backup drive on your PC

I mean, why rummage through your box of unused drives when it's too late?  Create a UFEI boot drive now, plug it into an unused (and fairly inaccessible) USB port on your PC, and you're ready to go whenever.  Makes backup and recovery a lot easier.

Taping the boot shortcut key for your PC to your monitor won't hurt, either.

So far, the boot environment of Macrium Reflect seems to work pretty well.

I don't like their file naming system, but I can live with it.

I haven't actually tried a bare-metal restore with it yet.

Tuesday, September 17, 2019

Blinking lights of death on constantly rebooting Synology Rackstation (or Diskstation)

Problem: Synology Rackstation (or Diskstation) reboots endlessly.  Power LED blinks, ALERT LED blinks.  Some drive LEDs green, most are red (or orange).  One drive LED might blink indicating drive activity.  LAN connection LEDs are dark (inactive).  No access. 

Solution:  Try powering it off, pulling all drives, peripherals and RAM, and rebooting.  (Don't forget any USB peripherals, like a UPS.)

If the Power/ALERT light behavior persists, and it doesn't return to a solid-on Power light in about a minute - your Rackstation is dead.

Or, at least, it took Synology support chat < 60 seconds to conclude mine was dead.  Maybe you will be more fortunate.


Synology would not confirm in my case, but I can't help but think the dreaded Intel C2000 bug could cause this.

BAD NEWS:  Unless your unit is in warranty, or covered under the Atom +1 year extended warranty, you are SOL.

GOOD NEWS:  Your data is most likely OK!  You can (carefully!) plug the drives in to a new NAS and all your stuff should still be intact.