I do. It can be faster, but for most CPAN modules, you won't notice a difference. I also have it enabled for my gentoo emerging - there it makes a bigger difference where there are files that really do take billions of cycles to compile.

On a scale of 1 to 5, 1 being "don't you dare" and 5 being "you should have done this months ago", I give it a solid "eh, whatever." :-)