Are your tests getting deeper into swap or virtual memory while running under 64 bit builds?

Under a 64 bit build of Perl, more memory is consumed because the architecture is twice as wide. A scalar on a 32 bit build may take considerably less of the system's memory than a scalar on a 64 bit build. Now apply that multiplier across large datastructures. If the memory footprint is driving your application into swap or virtual (storage-based) memory, you could start seeing large performance hits. The solution is, as always, more efficient algorithms or more hardware (in this case RAM).