joec_ has asked for the wisdom of the Perl Monks concerning the following question:

Hi,
I have a text file (300,000 lines) in which large chunks of data are seperated by $$$$ and a line feed i.e.

...
3 6 8 09
5 7 9 5
END
$$$$
5 9 0 -7
7 9 3 6
END
$$$$

I have a CSV file which contains data related to each group above. The first row in the CSV is a header row with column names.

My csv looks like

Column 1, Column 2
Data 1, Data 2
Data 3, Data 4

I would like to end up with an output file like:

...
3 6 8 09
5 7 9 5
END
> <Column 1>
Data 1
> <Column 2>
Data 2
$$$$
5 9 0 -7
7 9 3 6
END
> <Column 1>
Data 3
> <Column 2>
Data 4
$$$$

Basically, the csv data has been merged between the END keyword and $$$$. I have tried:

while (<DATAFILE>){

$string = $string.$_;
}

@individual = split /\$\$\$\$/,$string;

    foreach $i (@individual){
        print "inside outer foreach\n";
        foreach $h (@headers){
            print "inside header foreach - appending tags\n";
            @seperate = $i.">  <$h>\n";
            
        }
        @seperate = $i.'$$$$';
    }
    
    open (OUT, ">outtest.out");
    print OUT @seperate;

[download]

What this does is just print '$$$$'.

Any help appreciated.

Thanks - Joe.

@headers came from just reading the first line of the csv

Replies are listed 'Best First'.
Re: Appending data to large files
by moritz (Cardinal) on Jan 13, 2009 at 16:30 UTC
    This gets you into trouble: @seperate = $i.">  <$h>\n"; You're not pushing to the array, but rather assigning to it (and thereby clearing all previous values). Use push where appropriate.

    You'll also make your live easier by reading the file record by record: 

    local $/ = '$$$$'. "\n";
while (<DATAFILE>){
    # $_ contains a record here
}

    [download]
[reply]
[d/l]
[select]
Re: Appending data to large files
by kyle (Abbot) on Jan 13, 2009 at 16:36 UTC

    I'd use Text::CSV_XS to read the CSV file. I'd read the file you're trying to change a little at a time, looking for the delimiter, and insert the CSV records when I find it. 

    my $record_end = qq{\$\$\$\$\n};
while (<DATAFILE>) {
    next if ( $_ ne $record_end );
    # insert stuff from CSV here
}
continue {
    print OUT;
}

    [download]

    You don't have to read all of any file at once or even mess around with regular expressions.

[reply]
[d/l]
      Hi, when i try to use Text::CSV i get the following error: Can't locate auto/Text/CSV/getline.al in @INC

      although if i do $csv->version it works and displays the version so the module is installed correctly - any ideas?Thanks

      **UPDATE: I installed Text::CSV 1.10 and Text::CSV_XS 0.58 and now with this code: 

      $file = 'project_data.csv';
my $parser = Text::CSV->new();
while (my $row = $parser->getline($file)){

 @fields = @$row;
 }

      [download]

      I get the following error:

      Can't locate object method "getline" via package "project_data.csv" (perhaps you forgot to load "project_data.csv"?) at ./join_csv.pl line 25.

      Any ideas? Thanks

      **END**

[reply]
[d/l]

        The getline method is expecting an object with a getline method. Basically you have to open the file and pass in the handle instead of the filename. 

        my $csv_filename = 'project_data.csv';
open my $csv_fh, '<', $csv_filename
    or die "Can't read '$csv_filename': $!";
my $parser = Text::CSV->new();
while ( my $row = $parser->getline( $csv_fh ) ) {
    # work, work, work
}
close $csv_fh or die "close() failed: $!";

        [download]
[reply]
[d/l]
[select]
Re: Appending data to large files
by hbm (Hermit) on Jan 13, 2009 at 16:36 UTC
    Nit: s/seperate/separate/g
[reply]

Back to Seekers of Perl Wisdom