Somewhat OT answer

You need to parse "a bunch" of data. There seems to be real-time value in your data. Is it possible that a solution with a quick execution time is needed? This idea has led me to an off-topic answer, since it is really about how to make a quick program of the type that you describe.

One of the my favorite things about perl is the speed of the regular expression engine. It can parse lines very quickly by anchoring a match at the beginning of a line. The rest of the line can be parsed using a fast regular expression.

  
if (/^G017RATEBRKRL,([^,]+),([^,]+),([^,]+),(.*)/)
{
  $col[3]=$1;
  $col[5]=$2;
  $col[1]=$3;
  $col[4]=$4;
}
elsif (/^G017CP111 D,([^,]+),([^,]+),([^,]+),(.*)/)
{
  #  etc...
}
[download]

The negated character classes run quickly because they only need to look for commas.

Another perl speedup has to do with minimizing the number of copy operations needed to load a database with DBI. It should be possible to go from $1, $2, etc, into a data structure that can be directly loaded into the database, without being copied again.

It is great to be write programs that have wonderful abstractions in them. Sometimes it is even better to write programs that are wickedly fast.

It should work perfectly the first time! - toma