Hi,
I think that cpanm's "--configure-args" capability might be helpful to you here.
Check the cpanm documentation.

I rarely use any of the cpan* modules, but if you (alternatively) wanted to build modules manually, you could look at the ExtUtils::MakeMaker documentation for (eg) modifying CCFLAGS.
One thing that I've successfully used in manual builds is to modify the "CC" setting, so you might find that something like the following works:

perl Makefile.PL CC="cc -F /Library/Frameworks"
[download]

If that works, I'd probably just rebuild perl with the -Dcc="cc -F /Library/Frameworks" argument.

Cheers,
Rob