After a lot of fruitless searching and experiments I finally found the solution - can't believe how simple it is, and it was right here on this site.

export LDFLAGS=/usr/local/lib
perl Makefile.PL
[download]

Like LD_LIBRARY_PATH the LDFLAGS is only temporary, but it is only needed while making the lib.

Thanks for the help.