Does this explain it?

[me@host]$ perl -le 'print 1 if (0) or warn 3';
3 at -e line 1.
1
[download]

You see, it's the order of operations on the if and the or... it's taking the "or warn $!" in the directory-chmoding line, and executing it (if, of course, it's a file... you know, lazy evaluation and all). "warn $!" returns true... so the overall condition of (-d $_) or warn $! evaluates to rue overall. In the end, this causes all files and directories to have their mode set first to 644 and then to 755. The fix would be to do this, instead:

do { chmod($filemode, $_) if (-f $_) } or warn $!;
do { chmod($dirmode, $_)  if (-d $_) } or warn $!;
[download]