I think Unixish operating systems have a problem with determining the directory where an executable lives in and hence it's not really supported for a program to find out the directory it is living in to determine the place of library directories in relation to that path.

...but I think that's how it's meant to work, if I'm reading the docs correctly (from perl5100delta.pod):

Relocatable installations

There is now Configure support for creating a relocatable perl tree. If you Configure with -Duserelocatableinc, then the paths in @INC (and everything else in %Config) can be optionally located via the path of the perl executable.

That means that, if the string ".../" is found at the start of any path, it's substituted with the directory of $^X. So, the relocation can be configured on a per-directory basis, although the default with -Duserelocatableinc is that everything is relocated. The initial install is done to the original configured prefix.

So, I'd say the OP's expectations are at least somewhat legitimate :)