will that slow down processing?

Yes, it will be unnoticably slower (when import is called, that would be at compile time most of the time) than not exporting.

However, that's not where the slow part of your program is. Instead use a profiler such as Devel::NYTProf to find out where the real problems are, and optimise those.