I think the answer is probably "Yes, most of the time."

Perl releases are tested on a variety of platforms and (I imagine) are tweaked until all tests pass. Thus, a program that uses core Perl only should be architecture neutral.

Things may get interesting when CPAN modules are used -- I don't have any idea how they are tested on different platforms.