As a debugging suggestion: run the script twice, and then sort and diff the two resulting files.

Perhaps you can find patterns of what files are detected in one of the runs, and not in the other.

BTW which kind of file systems are you using? Any network filesystems?