Re: Adding cols to 3d arrays

First re the backup, and woeful lack of it in my case - this is so so true. So much effort could have been avoided if I had. The sad thing is that this is not the first time I have experienced significant data loss due to hard drive failure. In this case although I had previously used backups, I mistakenly thought SSDs were super robust, and had moved a load of new data (these photos mainly, a few thousand of them) from camera SD cards to the SSD while I organised it ready for moving to the backup machine. I then reused the SD cards :-( The 'organising' project took longer than expected due to family illness and I forgot the vulnerable data sitting on the SSD, thinking it 'safe as houses' anyway. Meanwhile the power supply developed a fault. Only then did I discover SSDs pitfalls.

Even so my old backup solution was, I can see now, not good enough. A fire or lightning strike would for instance destroy all my data, but also many other scenarios might. I now have a system which keeps 5 copies distributed over 2 locations several miles apart, using different OSes and formatting systems. One is usually offline. I think I still have holes in the system though and am looking to change a few aspects of it. I use spinning rust now, because it can usually be recovered from, certainly a lot easier than SSD. SSD tech I now realise is a complete nightmare, since a power failure at the wrong time can cause what I have - an extremely difficult, if possible at all, to recover drive.

As for optical media, I am wary of it, having had issues in the past with copies becoming unreadable several years later. Maybe poor quality as you say though. The other problem with it is I have around 3TB of to me very important data, once you figure in the video taken over the last few years, and optical with its small size takes time to use and make several copies of. I like HDDs now because they are very large, and as you say they are extremely mature. Data recovery companies usually have excellent success with them if required, the only severe failure mode is physical damage to the disk and with multiple copies in more than one location the chances of all having physical damage is very low. And they can be tasked to do their job reliably even when I am in periods of life when I am not being reliable!

A second storage technology would be nice but I wonder about the reliability of the higher density stuff. I must admit I didn't know some can hold up to 128GB before looking it up just now, and its something to look into certainly. A few optical backups on some media which can be trusted would be certainly nice to have.

Ill get back to the technical now in another post.

Comment on Re: Adding cols to 3d arrays - syntax

Replies are listed 'Best First'.
Re^2: Adding cols to 3d arrays - syntax by jcb (Parson) on Sep 22, 2019 at 22:08 UTC
There is one more advantage to optical media — it is the only commonly available format that is (or should be) entirely waterproof. Optical discs should retain data even after a flood — clean the mildew off and the disc reads fine. (Again, quality is important here, since poor quality discs might not be properly sealed, leading to "laser rot" even if not exposed to water.) This may or may not be relevant to your risk model, and 3TB is a very large amount of data. I have so far avoided the "unreadable several years later" problem by using good quality media from reputable manufacturers that I buy when the stores put it on sale (usually almost half-off if I am patient). Since I buy the blanks when they are on sale, I have a significant personal stock that I slowly rotate, and I suspect (and hope) that the blanks that will go bad will go bad before I get around to putting data on them. So far, this strategy has worked and I have yet to retrieve a disc from storage and find it unreadable, although I have had many discs fail verification immediately after writing them. Always read back an optical disc immediately after writing it — do not expect the drive to notice that the blank is bad while it is busy writing data. It is probably best to rank by importance (favoring more copies on lower-density media) and bulk (requiring fewer copies on higher-density media). This means the data with higher bulk-to-importance ratios (like high-def video) is exposed to greater risk of loss, but one partial mitigation is to store lower-resolution more-compressed copies of those videos in lower-density "bands" in your archive. I still use CD-Rs for some backups, even though I mostly use DVDs now. (But I do not have a significant collection of video.) So you might have full high-def video stored only on spinning hard disks, but lower-resolution "better than nothing" transcoded copies on DVDs or BDs. By now, you have probably learned better than to consider SSDs as valid backup storage. :-) (But they could still be a 3rd technology holding a 4th copy.)	[reply]

Replies are listed 'Best First'.

Re^2: Adding cols to 3d arrays - syntax
by jcb (Parson) on Sep 22, 2019 at 22:08 UTC

There is one more advantage to optical media — it is the only commonly available format that is (or should be) entirely waterproof. Optical discs should retain data even after a flood — clean the mildew off and the disc reads fine. (Again, quality is important here, since poor quality discs might not be properly sealed, leading to "laser rot" even if not exposed to water.) This may or may not be relevant to your risk model, and 3TB is a very large amount of data.

I have so far avoided the "unreadable several years later" problem by using good quality media from reputable manufacturers that I buy when the stores put it on sale (usually almost half-off if I am patient). Since I buy the blanks when they are on sale, I have a significant personal stock that I slowly rotate, and I suspect (and hope) that the blanks that will go bad will go bad before I get around to putting data on them. So far, this strategy has worked and I have yet to retrieve a disc from storage and find it unreadable, although I have had many discs fail verification immediately after writing them. Always read back an optical disc immediately after writing it — do not expect the drive to notice that the blank is bad while it is busy writing data.

It is probably best to rank by importance (favoring more copies on lower-density media) and bulk (requiring fewer copies on higher-density media). This means the data with higher bulk-to-importance ratios (like high-def video) is exposed to greater risk of loss, but one partial mitigation is to store lower-resolution more-compressed copies of those videos in lower-density "bands" in your archive. I still use CD-Rs for some backups, even though I mostly use DVDs now. (But I do not have a significant collection of video.) So you might have full high-def video stored only on spinning hard disks, but lower-resolution "better than nothing" transcoded copies on DVDs or BDs.

By now, you have probably learned better than to consider SSDs as valid backup storage. :-) (But they could still be a 3rd technology holding a 4th copy.)

[reply]