However, I'm slightly worried about the approach taken in Perl 6: everything is an object starting from fundamental types of data such as numeric constants. Sometimes it is useful to think in terms of objects, but often I want a number to be just a number, nothing more.

I personally believe you should not be worried, not even slightly. When you want a number to be just a number, you're free to think of it as such except for some situation in which you may need to call a method on it. But then you can even consider that as a little bit of funky syntax.