Load a csv file into a database (for example, mysql). The first line of the csv are the field names where data will be inserted (in order), and the rest, records to be inserted. It is similar to mysqlimport.
#!/usr/bin/perl -w use strict; use DBI; use Text::CSV; my $file = shift or die "Usage: $0 file.csv"; open CSV, $file or die $!; my @csv_content = <CSV>; close CSV; my $dbname = "dabasename"; my $dbhost = "localhost"; my $dbport = "3306"; my $dbuser = "username"; my $dbpass = "passwod"; my $table = "tablename"; my $dsn = "dbi:mysql:dbname=$dbname;host=$dbhost;port=$dbport;"; my $dbh = DBI->connect($dsn, $dbuser, $dbpass) or die "Connection erro +r: $DBI::errstr"; my $sth; my $statement; my $field_line = shift(@csv_content); chomp($field_line); my @values; my $status; my $line; my $csv = Text::CSV->new(); foreach(@csv_content){ if($csv->parse($_)){ @values = $csv->fields(); $status = $csv->combine(@values); $line = $csv->string(); $statement = "INSERT INTO $table($field_line) VALUES($line);"; print $statement."\n"; $sth = $dbh->prepare( $statement ); $sth->execute() or die "$! $DBI::errstr"; } }

Replies are listed 'Best First'.
Re: Load CSV file into Database
by Aristotle (Chancellor) on Jan 31, 2005 at 07:01 UTC

    You are preparing the same statement over and over. Use placeholders instead.

    If you pass DBI the RaiseError flag on connecting, you can avoid checking the success of each DBI operation manually.

    You also read the entire file into an array, then process it element by element. Read as you go instead.

    While we're at it, don't open the file yourself at all. Using the diamond operator to read from the files in @ARGV will handle errors automatically and lets you handle multiple files at no extract cost.

    There's also no provision in your script to pass the connection parameters in from the outside, so you have to change the source for every database it is supposed to run against. Look into Getopt::Long.

    #!/usr/bin/perl -w use strict; use DBI; use Text::CSV; use Getopt::Long; use constant USAGE => "usage: $0 [ -h hostname ] [ -p port ] --db dbna +me -u user -p passwd -t table file.csv [ file.csv ... ]"; GetOptions( 'h|host=s' => \( my $opt_host = "localhost" ), 'p|port=i' => \( my $opt_port = "3306" ), 'db=s' => \( my $opt_dbname ), 'u|user=s' => \( my $opt_user ), 'p|pass=s' => \( my $opt_pass ), 't|table=s' => \( my $opt_table ), ) or die USAGE; die USAGE if not( $opt_dbname and $opt_user and $opt_pass and $opt_table ); my $dbh = DBI->connect( "dbi:mysql:dbname=$opt_dbname;host=$opt_host;port=$opt_port", $opt_user, $opt_pass, { RaiseError => 1, PrintError => 0 } ) or die "Connection error: $DBI::errstr"; my $csv = Text::CSV->new(); my $sth = do { $csv->parse( scalar <> ); # first line my @field = $csv->fields(); my $sql = "INSERT INTO $opt_table (" . join( ',', @field ) . ") VALUES (" . join( ',', ( '?' ) x @field ) . ")"; print STDERR "Using query: $sql\n"; $dbh->prepare( $sql ); }; # # broken # while( <> ) { # $csv->parse( $_ ) or next; # $sth->execute( $csv->fields ); # } my ( $buff, $lnr ); while( <> ) { $buff .= $_; if( $csv->parse( $buff ) ) { $sth->execute( $csv->fields ); $buff = ''; $lnr = $. + 1; } } die "Error on line $lnr\n" if length $buff;

    Untested.

    Update: fixed code as per tilly's example.

    Makeshifts last the longest.

      Both your solution and the previous solution will silently omit lines with embedded returns. :-(

        Yeah, that is a limitation of Text::CSV. I chose to stick with the OP's choice of module anyway, but of course I should have said something about that.

        Makeshifts last the longest.

      For anything other than MySQL, you'll probably want to commit every so often, and also at the end.

      rdfield

Re: Load CSV file into Database
by jZed (Prior) on Jan 31, 2005 at 20:15 UTC
    Please use Text::CSV_XS instead of Text::CSV. The latter has not been updated in six years and the former is a) much faster and b) handles newlines.

    For DBMSs that have a LOAD command or loader utility, those propreitary solutions are likely to be better than a hand-rolled one.

    Aristotle's advice on placeholders and other improvements are really important.

Re: Load CSV file into Database
by bradcathey (Prior) on Jan 31, 2005 at 19:24 UTC

    Text::CSV or Text::CSV_XS is the way to go, but it can be done more conventionally, just in case the module is not available or you want to roll your own.

    .......connect to database....... .........upload CSV file .......... my $stmt ="DELETE FROM list" ; my $sth = $dbh->prepare($stmt); $sth->execute(); $stmt =qq/LOAD DATA LOCAL INFILE "..\/$csvtoupload" INTO TABLE list FI +ELDS TERMINATED BY "," OPTIONALLY ENCLOSED BY """" LINES TERMINATED B +Y "\r" IGNORE 1 LINES/; $sth = $dbh->prepare($stmt); $sth->execute(); unlink "../$csvtoupload" or die "Error deleting uploaded file: $!\n"; ...close db.....

    Of course, a bit more needs to be known about the original CSV, like does the first row need to be ignored.


    —Brad
    "The important work of moving the world forward does not wait to be done by perfect men." George Eliot