in reply to file state mannagment
Perhaps you could collect the names and last modification times of the available log files. You could safely process all but the latest ? Or are there multiple log files being written to at the same time ?