40GB is less than 10 DVDs worth of data. (less than 5 for dual-layer). With a little bit of up-front hardware cost, you can back up your collection, and take it off-site.

What you're discussing isn't quite as important with modern hard drives -- they're much better about checking for bad sectors, and the mean time between failures is much longer than the days of ~40MB disks. If you really wanted redundancy, you'd also want to deal with the failure of the disk controller, not just individual sectors, which would require moving to a RAID solution (or at the very least, mirroring your changes to another disk)

So, to answer your questions:

• 1. It depends on the hard drive mechanism. You'll have to look at the manufacturer's information.
• 2. Make sure there's a copy of the data elsewhere. (and if not a complete copy, parity information ... ie, something that can be used to recreate the data, such as what you're talking about)
• 3. If there is, I'd expect it to be in CPAN, and I'm not seeing it. There are modules if you search for ECC or CRC.

(yes, I know you can't rebuild the data from a CRC, but it would give enough to sense if something's gone wrong)

Thanks for the suggestions.

I do have backups on optical media. Note that DVDs are assumed to be much less reliable than disks, and are much less convinent for use anyway.

In the past 10 years this happened several times and so it hints that the drives are not that great. Making multiple copies is good, but would require a manual merging phase in case things go wrong. This is where a straightforward Perl script can come in handy.

BTW, if you back up to optical media, have a look at DVDisaster. They give a good background and a nice program which I have been using for my backups.

I know that DVDs don't have the longevity of magnetic media, but I should have mentioned a few things --

Never trust your backups. You should regularly refresh any important backups (ie, archival storage that you don't have a current copy of) ... I'd normally do a refresh at about 25-40% of the expected media lifetime, if I was only maintaining a single copy. (I'd move to 50% if I had two verified copies, written with different mechanisms, to different brands of media). There are no current standards for 'archival quality' DVDs. Based on one vendor, I wouldn't worry about media for 25+ years, depending on the media and proper storage.

For sake of argument -- magtape's recommended archival lifetime is 10-20 years. And they make a good point -- the bigger problem is change in technology. (ever had someone come to you with a 5 year old tape that no one bothered to keep a drive around to read? I've had it happen more than once (DDS1, 8MM video, DLT, reel-to-reel) ... sometimes the issue isn't the physical media, but the data can't be recovered because they didn't have the right software to restore it.

For the situation described (40GB backup) ... the media costs for mag tape is going to cost him more than the cost of a DVD burner ... and if he does DLT, he's going to need a SCSI or FC card ... I just can't see the justification for it given what was asked for. Hell, we're looking at backing up terabytes for long term archival storage (ie, speed to recover is not a major factor)

Thanks again for the long reply, but I feel that my question wasn't answered. What is it that you are refreshing? Could it be that you are making more copies of corrupt data? This can happen if your backup DVDs or hard-disks had some problem. For the hard-disk, this could also be something caused by a virus or mistaken overwrite.

I want to make absolutely sure that the data is fine. Also, for the case that it is not I want to be able to repair it. Making 2 or more copies does not solve that problem but luckily there are ECC.

I will post a script when its ready, as it will surely be useful for many people.

which specific DLT tape format were you unable to read later on a newer tape drive?
the hardest line to type correctly is: stty erase ^H

For only 40GB of data, buy abother disk and make a mirror or oven better, copy everything in DVDs.

Thanks.
I actually do have the data backed up more than once. Both on hard-disks and DVDs.

This doesn't help much though, since its enough that the file gets corrupted on the main copy, to propagate the corruption to the other copies. On occasion, I stumble onto a corrupt image and look it up in the old copies. This happened too many times before, on various disks.

I want to be absolutely sure that such cases do not occur since this data is precious for me.
Surely there is a better way.

i don't know if this would help or not

have you heared about MD5 CheckSum ?

you can copy all the images along with thier MD5 Keys

to a new HARD Disk

Check once per month for un-matched values and replace the currpted photo

Check if there's photos with out MD5 Key ... make a key for it and save it with other keys

BTW havent tryed to do this before ... just heared about it :P

and wanted to help

bye

Thanks for the suggestion. Using hashing is indeed good for checking data integrity. As you said, if a file is corrupt, one may hope to find a good copy elsewhere.

I want to achieve more than that. Using ECC, one can hope to be able to handle corruptions and so, a backup may still be good even if several bytes/blocks/sectors did go bad.

Storage is very cheap today. This means that increasing the size with redundant ECC information does not cost much. Another way to look at is that since storage _is_ cheap, one shouldn't count on it too much and devise a way to still maintain its reliability.