Well, since you're using Win32, you should get PSTools from SysInternals and get a "better" profile of when/where your app is using memory. I've whipped up a demo of what I'm suggesting:

C:\>perl -e "print grep(/Perl /,qx[pslist -m Perl])"
Name   Pid      VM      WS   WS Pk    Priv   Faults NonP Page PageFile
Perl  1632   28236    1416    1416     592      355    2   13      592

C:\>perl -MWin32::GUI -e "print grep(/Perl /,qx[pslist -m Perl])"
Name   Pid      VM      WS   WS Pk    Priv   Faults NonP Page PageFile
Perl  1516   34000    3200    3208    2680      802    3   17     2680

C:\>perl -MWin32::GUI -e "$main = Win32::GUI::Window->new(-name => 'Ma
+in'
, -width => 100, -height => 100);print grep(/Perl /,qx[pslist -m Perl]
+)"
Name   Pid      VM      WS   WS Pk    Priv   Faults NonP Page PageFile
Perl  1632   34000    3240    3248    2684      812    3   17     2684
[download]

By sprinkling pslist in your app and logging its output, you should get a better idea about what's going on. BTW, 'WS' is the number reported by Task Manager.

Update: Added column headings to the pslist output for clarity.

--
Don't be too proud of this technological terror you've constructed...