According to the top level Makefile.PL (for XML-Parser-2.36):

If expat is installed, but in a non-standard directory, then use the
following options to Makefile.PL:

    EXPATLIBPATH=...  To set the directory in which to find libexpat

    EXPATINCPATH=...  To set the directory in which to find expat.h

For example:

    perl Makefile.PL EXPATLIBPATH=/home/me/lib EXPATINCPATH=/home/me/i
+nclude

Note that if you build against a shareable library in a non-standard l
+ocation
you may (on some platforms) also have to set your LD_LIBRARY_PATH envi
+ronment
variable at run time for perl to find the library.
[download]

That's all I've ever needed to do. (Admittedly I haven't checked to see whether that advice is still current.)

Cheers,
Rob