What is going on here?

I tweaked your code by adding a couple of extra print statements:

my $mask = hex($ARGV[0]);

print $mask;
print unpack 'H*', $mask;

my $setbits = unpack("%32b*", $mask);

printf("0x%x has %d set", $mask, $setbits);
[download]

And when I run it I get this output:

C:\test>junk 0xff
255    ## 3 bytes
323535 ## 6 nybbles with bit patterns 0011 0010 0011 0101 0011 0101 
0xff has 11 set
[download]

hex converts the string '0xff' to a number which you store in $mask;

unpack expects a string, so perl helpfully converts the number stored in $mask to a string in the default decimal representation '255'.

You are counting the bits in that 3 byte string.

To count the bits in the number, you need to tell perl that you want the binary representation of that number:

#! perl -slw
use strict;

my $mask = hex($ARGV[0]);

print $mask;
print unpack 'H*', pack 'C', $mask;

my $setbits = unpack("%32b*", pack 'C', $mask );
printf("0x%x has %d set", $mask, $setbits);

__END__
C:\test>junk 0xff
255
ff
0xff has 8 set
[download]