Fellow Monasterians:

This might fall more into the realm of a MySQL forum, but I'll take a chance here in a familar place.

I wrote a little script that uploads a CSV file and imports it into a table. I do it all the time, but this time it is only importing a few lines. In a larger version of this same CSV file, it imports the first few lines, a couple in the middle, and then the last one.

I've examined it for "gremlins" and find the data clean. Frankly, I'm at a loss. Here's the statement that gets executed, or you can view the whole script below.

$stmt =qq/LOAD DATA LOCAL INFILE "..\/$cvstoupload" 
INTO TABLE statezones FIELDS TERMINATED BY "," OPTIONALLY 
ENCLOSED BY """" LINES TERMINATED BY "\r" IGNORE 1 LINES/;

[download]

Whole CSV file:

STATE,ABBREV,ZONE
Alabama,AL,2
Alaska,AK,4
Arizona,AZ,3
Arkansas,AR,2
California,CA,3
Colorado,CO,2
Connecticut,CT,2
Delaware,DE,2

[download]

All that imports:

Alabama,AL,2
Alaska,AK,4
Arizona,AZ,3

[download]

Entire script:

#!/usr/bin/perl -T

use lib "/home/gotjunk/www/cgi-bin";
use warnings;                                
use CGI::Carp qw(fatalsToBrowser);    
use strict;
use CGI;
use Crypt::CBC;
require Common;

#------------------------- declare --------------------------

my $query = new CGI;
my $dbh = &dbconnect; #open DBI connection in Common

#------------------------ grab input ------------------------

my $cvstoupload = &upload_file($query->param('filename'));

my $stmt ="DELETE FROM statezones"; #dump the old data
&execute_it($stmt);

#----------------- open .db file and insert -----------------

$stmt =qq/LOAD DATA LOCAL INFILE "..\/$cvstoupload" 
INTO TABLE statezones FIELDS TERMINATED BY "," OPTIONALLY 
ENCLOSED BY """" LINES TERMINATED BY "\r" IGNORE 1 LINES/;

my $sth = $dbh->prepare($stmt);
$sth->execute();

unlink "../$cvstoupload" or die "unlink: $!";

$sth->finish();
$dbh->disconnect();    
exit();

# ----------------------- upload file -----------------------

sub upload_file {

   $| = 1;
   my $sourcefile = shift;
   my ($buffer, $bytes);

   $sourcefile =~ /([\w .-]+)$/i;
   my $newfile = $1;
   $newfile =~ s/ //;

   open (OUTFILE, ">../$newfile") 
      or die "Cannot open $newfile: $!";
   binmode(OUTFILE);

   while ($bytes = read($sourcefile, $buffer, 1024)) {
      print OUTFILE $buffer;
   }
   close(OUTFILE) or die "Close: $!";
   chmod (0666, ">../$newfile");

   return ($newfile);
}

[download]

Thanks in advance

UPDATE: After testing all conditions and not finding anything in particular, I rebuilt the table and CSV file and got it to work. Thanks to those who responded.


—Brad
"Don't ever take a fence down until you know the reason it was put up." G. K. Chesterton

In reply to Unable to load entire CSV file into DB by bradcathey

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.