Brian L. Matthews explains it here as follows:

1. A reference is not a memory address. If you go to the memory address corresponding to the scalar value of a reference you will not find your data there.
2. A reference contains type information. A mere memory address, on the other hand, contains no type information and thus can store any object.
3. Larry Wall intentionally decided to use the term "reference" rather than "pointer" so that people would understand that Perl is not pointing to a memory address.

As Larry points out, Perl is not appropriate for all applications. For example, Perl itself is not written in Perl, it is written in C.

If you have an application which requires low-level manipulation of the contents of memory addresses then that application is not an appropriate candidate for coding in Perl.

Furthermore, it says in the Learning Perl book by our own Merlyn et al that a reference is somewhat like a pointer, but it is safer. Presumably one reason that a reference is safer is that one cannot easily manipulate references to poke holes that can be exploited in an operating system.