I'd say the biggest problem with using closures is that the number of people who fully understand them and can comfortably deal with code using closures is very small -- even smaller than the number who can comfortably deal with OO code. If you spend time on #perl or certain PM threads, it seems like everyone knows this stuff, but in reality it raises the bar quite a bit. You may have a hard time finding a module maintainer or an employee who can deal with closure-heavy code. I say this after having plenty of trouble trying to hire people who can deal with what I consider basic OO code.
I'm not saying there are no good uses for closures, but they aren't something I'd use frequently.