Treating a reference as a number would only be a problem if
it were possible to create (i.e. compute) a new reference
numerically. This would be equivalent to allowing pointer
arithmetic. As long as this is not the case, one
representation for comparison purposes should be as good as
any other, as long as there's a one-to-one correspondence
between the references and their representations.