I used some print statements in the loop to find the last dn it processed, then when I immediately looked after it I found the recurring non-complaint ldif comment showing up. So now I process it with perl to remove the comments before reading it into ldif module.