So it seems the free is working but its releasing the memory back to perl and not to the system .. If I'm wrong and it should go back to system let me know.

:) Its S.O.P., memory is returned to the system at perl's discretion, see Mini-Tutorial: Perl's Memory Management and perlclib. Perl probably overrides/redefines free/malloc (: and then swig does stuff -- I don't swig :) but you should probably stick with New/Safefree as perclib recommends