SH!T. It seems there is no problem anywhere in the code. I just browsed through my tab delimited text file. There are tab delimited entries of length 26000 characters. When I removed them, then there is no memory build up.
Somehow an entry in an array of size 26000 characters messes up the memory somehow.