I don't have a correct answer in hand, though if I were faced with this problem, I'd look at what server log rotation tools do. They face a similar problem.