I'm sorry that I didn't make myself clear. The regex above just is $1 and $2 because I deleted the references up to $31. Reading in 1 line at a time with perl just takes too long (1.5 million lines - 14 to 20 minutes per file) and awk can do it in like 2 minutes. I'm trying to cut the 14 minutes down to ~ 2 minutes as much as possible.
Well, to speed up your regex as much as possible, you must
make it so that there is as little possibility for backtracking
as possible. Try something like:
my $r = join "[|]" => ("([^|]*)") x 31;
while (/^$r\n/mg) { ... }