True enough. Currently, I'm leaning towards using
pkg-config in Makefile.PL to test for "outside-perl" libraries and compiler flags. Ofcourse that means another dependency, but at least you can easily croak out of Makefile.PL if you can't run pkg-config.
I'm not sure if there are better ways to do it.