Last time I compiled Perl for a 64 bit machine I had real problems that arose from the fact I initially tried to build a 64bit version with a 32 bit version of gcc. Are you sure that the files created by your gcc are 64bit?

I tested by creating hello.c, compiling it and runing the file command on the result.

I found that having a 64 bit version of gcc which created 64bit executables by default made all the issues go away.