illc0mmunication has asked for the wisdom of the Perl Monks concerning the following question:

I am having importing a CSV file using into MySQL. I'm hoping perl can help me resolve this by either reformatting the CSV file or doing some other magic. The file contains memo fields which terminate several lines, but are part of the same record. ex.

ID,Name,Memo
"1","John Smith","Memo Line1
Memo Line2
Memo Line3"
"2","Jane Doe","Memo Line1
Memo Line2
Memo Line3"

There are also cases where there are null fields represented as:

ID,Name,Number,Address,Memo
"1","Jane Smith","","","Memo1
Memo2"

I've tried using sed to do this, but it seems to get confused when it comes across some of the null fields i.e.
""

Basically, I'd like to reformat in a way that I can terminate each record with a set of characters, something like:

ID,Name,MemoEOL
"1","John Smith","Memo Line1
Memo Line2
Memo Line3"EOL
"2","Jane Doe","Memo Line1
Memo Line2
Memo Line3"EOL


Any help will be greatfully appreciated. And anyone who gets me a good answer might get something from their CDNOW wishlist! :)

Replies are listed 'Best First'.
Re: Parsing CSV file
by sauoq (Abbot) on Aug 10, 2002 at 16:42 UTC
    Text::CSV and/or Text::CSV_XS will help you get started. Oh... and if you are using the DBI anyway, don't forget DBD::CSV.
    -sauoq
    "My two cents aren't worth a dime.";
    
      And to throw in one more module, Text::xSV, which was written by our own tilly

      TStanley
      --------
      Never underestimate the power of very stupid people in large groups -- Anonymous
Re: Parsing CSV file
by DamnDirtyApe (Curate) on Aug 10, 2002 at 17:53 UTC

    DBD::CSV can see past the line breaks, and will read the Memo field as you intended. I think your best bet is to use the DBI all the way... Something like this:

    #! /usr/bin/perl # Untested, but ought to be close, if not right on. # Assumes CSV file is called 'people', in /home/me/db. use strict ; use warnings ; $|++ ; use DBI ; my $dbh = DBI->connect( "DBI:CSV:f_dir=/home/me/db;csv_eol=\n", { RaiseError => 1 } ) ; # Get the data out of the CSV file. my $sth = $dbh->prepare( 'SELECT * FROM people' ) ; $sth->execute ; my $data = $sth->fetchall_arrayref( {} ) ; $dbh->disconnect ; $dbh = DBI->connect( "DBI:mysql:my_db", 'username', 'password', { RaiseError => 1 } ) ; # Dump into the MySQL table. $sth = $dbh->prepare( <<'END_OF_SQL' ) ; INSERT INTO my_people ( ID, Name, Memo ) VALUES ( ?, ?, ? ) END_OF_SQL foreach ( @$data ) { $sth->execute( $_->{'ID'}, $_->{'Name'}, $_->{'Memo'} ) ; } __END__

    _______________
    DamnDirtyApe
    Those who know that they are profound strive for clarity. Those who
    would like to seem profound to the crowd strive for obscurity.
                --Friedrich Nietzsche
Re: Parsing CSV file
by grep (Monsignor) on Aug 10, 2002 at 22:28 UTC

    This tool has already been written (and perfected) and most importantly it's sitting on your drive with mysql.

    mysqlimport is a standard mysql utility.

    It takes just a few minutes to read throught the man page for mysqlimport and setup the command to import your CSV file. Then you can spend you valuable time writing perl code :)



    grep
    Just me, the boy and these two monks, no questions asked.
      DBD::CSV worked like a charm, don't know why I didn't see it before. DUH!

      I first tried mysqlimport, but it has problems with the line breaks in the memo field.

      Thanks everyone for your help.