One more idea. Could you be suffering from a quota/limit on open file descriptors? Try making sure that all files you have finished processing are properly closed.