in reply to rsync workalike

I don't have too many suggestions. One problem is that you are getting very specific about what you want to implement, but you aren't explaining the scope of what you are actually trying to do.

For example, when I think of the term "Backup" I don't tend to think "rsync" since backups tend to keep multiple versions of a file, and the files eventually tend to be aggragated onto cheaper tertiary storage (like tape) for scalability/price reasons. Storing these as individual objects on the servers disk and/or in a database is not necessarily efficient (i.e. not "low on resource usage"). Another example is that you aren't defining how many files a server will handle, how many clients per server, etc.

If you really need something like rsync, one thing you might try is to bundle up many small files at the client end into a single reasonably sized group (e.g. tar) and work with that as a unit. Depending on the file sizes you are dealing with this could cut the number of database entries, and/or checksums down by a factor of 100 or more. Another nice thing about this is that you are forcing the client to do most of the work -- the server doesn't need to stat 100 files. If a small file changes on the client, you could recreate the entire bundle since the total size of the bundle would be reasonable.

FWIW, I've heard someone creating fairly scalable bundles similar to this by just bundling files in the same directory (i.e. don't recursively descend the tree so that there is 1 bundle per directory). One nice thing about this is that the bundle file has the same parent directory path as every file in it, so you don't have to play games trying to hunt down which bundle a file is located in.

Replies are listed 'Best First'.
Re^2: rsync workalike
by NiJo (Friar) on Aug 12, 2004 at 21:29 UTC
    Thank you very much for your suggestions. I was more after help on replacing rsync than my application. It was only ment to serve as background. Simplified I want to do something like:

    rsync-copy --source /path/file --template remote:/pool/<old_checksum> --target remote:/pool/`md5sum /path/file`

    for many files. The goal is to create a centralized smart backup application (e. g. for home use). Think of a full backup as fast as "updatedb" on Linux/Unix. It has to work across low bandwith links, e. g. analog phone lines. I want to exploit the fact that many clients share most data in different locations. That's mostly operating system and applications. Clients are too dissimilar for simple imaging. They share 90% of data across the same OS distribution.

    I don't have many details about file count on server (1e6 different checksums?), number of clients (1e3 ?) or distribution of file sizes. Despite the large numbers, server DB scalability, performance and disk space should not be a major issue. One DB on the client scales well. Of course the client needs to do 99% of the work.

Re^2: rsync workalike
by adrianh (Chancellor) on Aug 15, 2004 at 13:43 UTC
    For example, when I think of the term "Backup" I don't tend to think "rsync" since backups tend to keep multiple versions of a file,

    There's actually quite a nice technique using rsync for disk based backups that takes advantage of Unix hard links to do fairly efficient multiple revisions.