Unfortunately, you generally can't mix 32-bit and 64-bit code in one process, which means that a 64-bit Perl binary (or the respective XS code) simply isn't able to load a 32-bit shared library.  In other words, there is no (or no sensibly easy) way around having to use a 32-bit Perl.