As someone else mentioned, if you have $$ to spend, check out SyncSort. They should be able to do everything you're looking to accomplish. I've never actually done all those things myself with SyncSort, but most of them. After it's installed and configured, regular maintenance was pretty simple.