While I don't know what specifically is wrong, I am somewhat dubious about trying to compile Perl with a cross-compiler. One of the things it has to do is probe your system to find out what you have available. If you are cross-compiling it is unlikely to find the right things in the right places, and if you are compiling for a different architecture, it is unlikely to be able to link the libraries it does find (which are for your current architecture) with the code you are trying to compile (for your target architecture).

Therefore I would try to compile on your target architecture.

OTOH this is something I haven't tried to do. If you really need to do it, then I would suggest asking the main development list for ideas and pointers.