Looking at the XS code for Win32::Dirsize it looks like it is doing more or less the same thing as you are. OK, so it is using C, but most of your work is being done at a low-level as well, so I'm not sure that you would see much of a performance improvement.

If you right-click on a directory name on Windows Explorer and select 'Properties' you will find that it takes quite a while to get the "size on disk" the first time you do it. Is your Perl substantially slower than Windows Explorer?