file: line(s) /\z/
line: "G017RATEBRKRL" comma rate comma start_date comma end_date comma
+ time newline { ... }
line: "G017CP111 D" comma start_rate comma end_rate comma change comma
+ date comma time newline { ... }
line: "G017RPAGO/N" comma rate comma whitespace  comma whitespace  com
+ma date comma time newline { ... }
[download]

-- Randal L. Schwartz, Perl hacker

startrule : G017RATEBRKRL | G017CP111_D | G017RPAGO_N | G017ONFD | G01
+7PDFF
[download]

G017RATEBRKRL_Key : "G017RATEBRKRL"
G017RATEBRKRL : G017RATEBRKRL_Key type comma rate comma start_date com
+ma end_date comma time { return \%item }
[download]

As for the commas, I don't think there's a way to get rid of them easily. Punctionuation typically has to be specified in lex grammars, and that's what you're doing here.

-----------------------------------------------------
Dr. Michael K. Neylon - mneylon-pm@masemware.com || "You've left the lens cap of your mind on again, Pinky" - The Brain
It's not what you know, but knowing how to find it if you don't know that's important

use Parse::RecDescent;
use strict;
use warnings;
use Data::Dumper;

# Make sure the parser dies when it encounters an error
$::RD_ERRORS = 1;
# Enable warnings. This will warn on unused rules &c.
$::RD_WARN   = 1;
# Give out hints to help fix problems.
$::RD_HINT   = 1;

# Create and compile the source file
my $parser = Parse::RecDescent->new( q(
    comma      : ","
    date       : /\b\d{1,2}\/\d{1,2}\/\d{1,2}\b/
    start_date : date
    end_date   : date
    time       : /\b\d\d:\d\d:\d\d\b/
    rate       : /\b\d+\.\d{4}\b/
    rates      : rate comma { $item{rate} }
    start_rate : rate
    end_rate   : rate
    change     : rate
    whitespace : /\s*/

    lines : line /\z/ { $item{line} }
    line  : "G017RATEBRKRL" comma rate comma start_date
      comma end_date comma time { $item{type} = $item[0]; \%item }
    line  : "G017CP111 D" comma start_rate comma end_rate comma
      change comma date comma time { \%item }
    line  : "G017RPAGO/N" comma rate comma whitespace comma
      whitespace comma date comma time { \%item }
    line  : "G017ONFD" comma rates(6) date comma time { \%item }
    line  : "G017PDFF" comma rates(4) date comma time { \%item }
) );

while ( my $quote_data = <DATA> ) {
    next if $quote_data !~ /\S/;
    my $result = $parser->lines( $quote_data );
 
    if ( defined $result ) {
        print Dumper $result;
    } else {
        print "Failed!\n";
    }
}
[download]

You wrote:

As it turns out, each line of the file represents one type
of quote and the format and the format, while consistent for 
each quote type, varies from type to type. In other words,
one line may have five fields and the next line may have
eight.
[download]

It seems to me that by the time you've done something like

while(<DATA>)
{
  chomp;
  my @line = split(',',$_);
}
[download]

It seems to me that anything from @line[1..(@line-3)]
is the data you are interested in. This is one way to deal
with variable-length fields in your data:

#!/usr/bin/perl

while(<DATA>)
{
  chomp;
  /^$/ and next;
  my @line = split(/,/,$_);

  for(@line[1..(@line-3)])
  {
    /\d+\.[\d]{4}/ or next;
    printf("%s\t%s\n",$line[0],$_);
  }

}

exit;
[download]

I haven't done any of the nifty stuff you are doing-
building references etc.- because I still have a hard time
grokking them...

Tentatively,
blyman

This is a clue; either something is wrong with your actions or the grammar isn't parsing the data correctly. Try adding | <error> clauses to the end of each top-level rule. This will tell you if there's a parsing error, and possibly what the error is. If this doesn't show anything, look hard at the actions. You may want to explicitly set the $return variable in the actions.

By the way, Parse::RecDescent is kind of overkill for something like this. P::RD is made for much more complicated things (I'm working on using it to parse a subset of Perl) than several stock quote formats; this may be why it seems so clumsy. However, that doesn't mean "don't use it"--it just means "look at alternatives".

=cut
--Brent Dax
There is no sig.

You need to parse "a bunch" of data. There seems to be real-time value in your data. Is it possible that a solution with a quick execution time is needed? This idea has led me to an off-topic answer, since it is really about how to make a quick program of the type that you describe.

One of the my favorite things about perl is the speed of the regular expression engine. It can parse lines very quickly by anchoring a match at the beginning of a line. The rest of the line can be parsed using a fast regular expression.

  
if (/^G017RATEBRKRL,([^,]+),([^,]+),([^,]+),(.*)/)
{
  $col[3]=$1;
  $col[5]=$2;
  $col[1]=$3;
  $col[4]=$4;
}
elsif (/^G017CP111 D,([^,]+),([^,]+),([^,]+),(.*)/)
{
  #  etc...
}
[download]

Another perl speedup has to do with minimizing the number of copy operations needed to load a database with DBI. It should be possible to go from $1, $2, etc, into a data structure that can be directly loaded into the database, without being copied again.

It is great to be write programs that have wonderful abstractions in them. Sometimes it is even better to write programs that are wickedly fast.

It should work perfectly the first time! - toma

Maybe I'm not totally understanding the problem here, but are you committed to Parse::RecDescent for any reason other than it may do what you want?

If all you need to do is read in CSV files where each file contains its own arbitrary format, perhaps the AnyData module would be sufficient for your needs???

While I agree that using Parse::RecDescent may be overkill, particularly in this day and age of XML-based file formats, it's still a good idea if you want to be a jack-of-all-trades to have understanding and ability to do grammar parsing; at some point, if you are building custom applications, you'll undoubtable come across a grammar-based format that no other parsing method would easily work with. My impression from Ovid's request was that while there were many other ways, possibly simpler, to do this, learning P::RD was a subgoal of this solution.

-----------------------------------------------------
Dr. Michael K. Neylon - mneylon-pm@masemware.com || "You've left the lens cap of your mind on again, Pinky" - The Brain
It's not what you know, but knowing how to find it if you don't know that's important