Is there anyway out make to return that unused memory to OS?

Write your own malloc() and configure Perl to use that. It's not impossible, but there are a lot of niggly little bookkeeping details, especially if you want to avoid fragmentation.