Your real speed loss is going to be from the spawning of a new process via the backticks, not the actual number conversion. As blazar also noted, you can use Math::BigInt within the scope of a specific block. So something like this will work (and this could easily be put in a loop):

 perl -le '$x = do {use Math::BigInt qw/:constant/; 2**96; }; print $x
+; print 2**96'
[download]