You remember wrong. It's all explained in perlobj. I quote:

A clarification: Perl objects are blessed. References are not. Objects know which package they belong to. References do not. The bless() function uses the reference to find the object. Consider the following example:

        $a = {};
        $b = $a;
        bless $a, BLAH;
        print "\$b is a ", ref($b), "\n";
[download]

This reports $b as being a BLAH, so obviously bless() operated on the object and not on the reference.