Synology Aggravation

I've read of this happening to someone before. Not sure what to make of it, beyond aggravation.

We had a power outage last week. I have three NAS boxes, all Synology. They're on UPS, so when the power went out, I had time to give them an orderly shut-down.

Regardless of that (and I've had to shut them down before for various reasons), when I restarted one box, it began beeping and complained that one of the four drives was severely degraded and required replacement.

Well, bother!

But this was the reason I use a NAS with RAID 5 and hot-swap capability. So I hopped down to our local MicroCenter and bought a replacement drive. Popped it in.

Then, after a night integrating that replacement drive, the NAS unit complained that another drive was degraded and needed immediate replacement. What? It was fine before. So another replacement and another night of integration.

That happened a third time, too.

I'm going through some tests to determine if the drives really are degraded as the Synology box claims, but I do have to admit that the system was able to accommodate a new drive a day without losing data.

These were WD red drives, btw.

I have had a two-drive Synology configured as RAID1, since about 2017/18. Have had sudden interruptions of power (power disrumptions & inadvertant unplugging!) occur with no undesireable results. I couple years ago, I replaced one drive with a larger capacity one, and let the RAID1 make a mirror copy onto the new drive, and then I replaced the second drive with a larger one and again let the RAID1 rebuild that drive. Been on the new larger drives for a couple years now, still keep the old ones for all the older data already on them, for extra data redundancy since all their contents are on the new drives, too. No problems ever induced by unexpected loss of power, even though I do not use an UPS.

That's typical for a lot of large RAID rebuilds. In the clusters that I maintain (big enterprise RAID boxes), I don't run the same manufacturer/make/mode​l for all drives as it's common for drives of a batch to fail around the same time due to a flaw, design issue, or other issue. RAID 5 is well known for your issue - a second or subsequent drive failing during rebuild of another. The stress of the hard, constant read to rebuild the parity stripe can stress out other weak drives and cause them to fail. Many a R5 has been lost over the years because of the issue. Back in the day when drives were < 1TB, R5 was ok. With the larger drives and extended rebuild times, nobody runs R5 anymore - it's too risky. It's at least an R6 but typically R50+1 or R60+1 for true data integrity. For backups, it's mostly just large R1 with big drives.

Western Death is well known too for their consistent drive failures after one fails, especially for the cheaper consumer drives, like the Red.

The other issue is that Synology and other consumer NAS boxes usually don't live test the drives, but rely on SMART data to calculate/predict impending failure. Many times, SMART doesn't record any major issues until it's been shut down. Then, during startup, the SMART data is checked and that's when it gets noticed. There's also a grey area where the firmware predicts possible failure inbetween the "normal" and "failing" states. That threshold isn't always reported and errors in that grey area don't always trip alarms. When the drive massively exceeds errors or has numerous in a short period of time, it then trips the alarm.

In your situation, it may have have run for months more before the warning picked up on that first drive. Or, the second drive may have run for months more if it didn't need to rebuild and drove the drive hard. It's actually safer nowadays to run 2x 20TB in R1 (with one being a different brand than the other) than it is to run 5x 5TB in R5. You're 1) less likely to encounter a rebuild error resulting in total array failure, 2) get faster performance, and 3) have faster rebuild times as it simply has to copy to the new drive instead of rebuilding from parity striping.

Synology, Areca, QNAP, etc are all good devices. It's the drive quality and inherent redundancy technology that usually bites you. We don't run "NAS grade" drives - we use enterprise grade drives that are designed to be driven hard. We also run higher RAID designs to ensure rebuildability. The last thing is to routinely test the drives AND have a preemptive replacement schedule where you roll out old drives before expected failure. That's based on the usage of the cluster and warranty of the drive. Ignore the "MTBF" number - that's a statistical guess on the average drive life. If there's a 3 year warranty on the drive and it's a heavily used cluster, we start to pull drives around 2.5 years in. For a lighter used cluster, we may run those drives up to 4 years, if the system has a +1 or +2 hot standby ready to roll in the bay.