When you profiled the code where did the time get spent? Take a look at Devel::Profile if you are not using it already. If your code really is slow (you've not actually told us how long it takes - really "hours"?) the time you spend learning to use the profiling tool will pay for itself in short order.

As an aside, are you sure your tests are correct? I'd be a bit worried about using $2 like that, especially as you use further matches after the first match.

It would help if you showed us a sample of your input file so we can see its structure and maybe suggest a better way to process it. I suspect a single pass through the lines should suffice, but we can't tell from the data you've not shown us.

Almost always the big time wins come from changing your algorithm rather than fine grain tweeking of existing code.