Re: Alternatives to DB for comparable lists

complete integrity against bit rot is needed

Consider using a filesystem that guarantees exactly that. ZFS does, when using several disks, and can be considered stable. It also offers deduplication, snapshots, transparent compression, replication across machines, and more.

btrfs attempts to do the same, or at least some of the important parts, but I would still call it experimental.

If you want a no-brainer setup, find a x86_64 machine with RAM maxed out, add some disks, and install FreeNAS or the fork NAS4Free. It natively uses ZFS and doesn't force you to think much about it. Add a similar second machine and use replication for backup.

Alexander

--
Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)

Comment on Re: Alternatives to DB for comparable lists

Replies are listed 'Best First'.
Re^2: Alternatives to DB for comparable lists by peterrowse (Acolyte) on May 16, 2018 at 00:10 UTC
I am already a convert - the destination is ZFS, its great. Problem is all this legacy stuff, spanning many years, many times moved / duplicated from disk to disk etc, although its all on EXT4 now, which seems pretty good, but in the past... So at times I am going to need to work out which are the damaged versions manually. But I hope to hone it down to a few files with the rest all matching from version to version (hopefully a few that is, we will see).	[reply]

Replies are listed 'Best First'.

Re^2: Alternatives to DB for comparable lists
by peterrowse (Acolyte) on May 16, 2018 at 00:10 UTC

I am already a convert - the destination is ZFS, its great. Problem is all this legacy stuff, spanning many years, many times moved / duplicated from disk to disk etc, although its all on EXT4 now, which seems pretty good, but in the past... So at times I am going to need to work out which are the damaged versions manually. But I hope to hone it down to a few files with the rest all matching from version to version (hopefully a few that is, we will see).

[reply]