For example, when I think of the term "Backup" I don't tend to think "rsync" since backups tend to keep multiple versions of a file, and the files eventually tend to be aggragated onto cheaper tertiary storage (like tape) for scalability/price reasons. Storing these as individual objects on the servers disk and/or in a database is not necessarily efficient (i.e. not "low on resource usage"). Another example is that you aren't defining how many files a server will handle, how many clients per server, etc.
If you really need something like rsync, one thing you might try is to bundle up many small files at the client end into a single reasonably sized group (e.g. tar) and work with that as a unit. Depending on the file sizes you are dealing with this could cut the number of database entries, and/or checksums down by a factor of 100 or more. Another nice thing about this is that you are forcing the client to do most of the work -- the server doesn't need to stat 100 files. If a small file changes on the client, you could recreate the entire bundle since the total size of the bundle would be reasonable.
FWIW, I've heard someone creating fairly scalable bundles similar to this by just bundling files in the same directory (i.e. don't recursively descend the tree so that there is 1 bundle per directory). One nice thing about this is that the bundle file has the same parent directory path as every file in it, so you don't have to play games trying to hunt down which bundle a file is located in.
In reply to Re: rsync workalike
by bluto
in thread rsync workalike
by NiJo
| For: | Use: | ||
| & | & | ||
| < | < | ||
| > | > | ||
| [ | [ | ||
| ] | ] |