Maybe I should have mentioned several facts I omitted in the original post, sorry. Here comes some background information:
- The numbers reported were not single measurements, but resulted from an NYTProf run with the script processing the 1st 100 lines of a template of the file under consideration. I consider this to be enough for getting halfway stable results
- Any of the lines processed contains about 30.000 fields. With this number the difference does matter, belief me. In particular as the regular final input file size in production has a range of between 1e4 and 1e5 fields and around 1e6 lines. Thus even saving 10% or 20% of runtime is worth the effort
- Thanks for the suggestion of the Benchmark module. Nonetheless I'd rather not repeat the processing 500.000 times as this would mean about 12 days only for line splitting with limited gain of knowledge ;-)
In production with external time measures at least processing seems faster with global than lexical variables.
BTW:
Using C or Pascal instead and coding splitting, filtering and joining by hand does not seem to gain much here (to be honest: almost nothing), not to mention using provided libraries, which seem to be even slower as they favor generic solutions. This only costs more of programmer's time.