One way to do it is to open up the Makefile.PL, find the call to WriteMakefile and change the LIBS parameter: add the full path to the library:

WriteMakefile(
  ...
   LIBS => ['-L/some/path', '-lexpat' ... ],
  ...
);
[download]

There are probably better ways, though.