After experimenting lots of solutions and code changes I was finally able to fix the issue.

As evident from the beginning the issue was with sharing SSLeay (and Mechnaize) between multiple threads. I was sharing Mechanize object as I needed the session to be maintained.

Solution came very simple. Save the cookies after login and use the cookies to resume last session while creating new mechanize object. Very basic isn’t it.
Now I don't share Mechanize agent instead I share the cookies and create a agent per crawler