in reply to mySQL Times Out / Disconnects
Also, you might save some overhead by preparing your sql statements with placeholders. (And if you have names like "O'Toole", using placeholders will save you a lot of grief.)
If your "Execute" function (wherever that is coming from) doesn't support placeholders, don't use it -- just go right to the standard DBI functions. Here's technique I've used to good effect on several occasions, with hashes to hold the sql statements and statement handles:
(updated to fix indents; updated again to add the missing "for (keys %sql)" to populate the %sth hash.)# assuming that the csv file has been opened and read into @CSV_FILE $dbh = DBI->connect("DBI:mysql:DBname","$un","$pw") or Terminate("Sorry, Disconnecting: $DBI::errstr"); my %sql = ( get => "SELECT id, lastname, firstname, ssn FROM member". " WHERE UPPER(lastname) = ? AND UPPER(firstname) = +?", ins => "INSERT DELAYED INTO reports (lastname, firstname, +title, ID)". " VALUES (?,?,?,?)", upd => "UPDATE LOW_PRIORITY member SET working = 'Y'". " WHERE ID = ? LIMIT 1", ); my %sth; $sth{$_} = $dbh->prepare( $sql{$_} ) for ( keys %sql ); for my $csv_row(@CSV_File) { my ($FN, $LN, $SSN, $Title) = @$csv_row; $sth{get}->execute( $LN, $FN ); my $member_rows = $sth{get}->fetchall_arrayref(); for my $db_row ( @$member_rows ) { my ( $id, $ln, $fn, $sn ) = @$db_row; if ($SSN eq $sn) { $sth{ins}->execute( $ln, $fn, $Title, $id ); $sth{upd}->execute( $id ); &LogEvent("Success"); } } }
But you can probably retool this even further: the CSV file has three fields for identifying people: first_name, last_name and "ssn". Those three are also in the "members" table, but you select on the basis of first and last name, then reject a row when the ssn doesn't match.
Why not select using all three fields in the first place?
If a row comes back, great -- do the update and insert for that person. If not, report that as an error (probably a typo in the csv file, or maybe a row in the members table is wrong/missing; either way, the log message should include the name along with the ssn from the csv file).get => "SELECT id FROM member where". " ssn = ? and UPPER(lastname) = ? AND UPPER(firstname) = ?",
BTW, are you sure the csv file has names in all-upper-case?
I don't know if these suggestions will help with the problem you are actually having, but if they don't, you should check out whether you really need the "DELAYED" and "LOW_PRIORITY" modifiers.
You might also look into doing the inserts by writing the rows to a file as tab-delimited lines, and then doing "LOAD DATA LOCAL INFILE" (add LOW_PRIORITY if you want) after you have finished looping over all the csv rows. When you're doing thousands of inserts, that can save a lot of time.
|
|---|
| Replies are listed 'Best First'. | |
|---|---|
|
Re^2: mySQL Times Out / Disconnects
by JayBee (Scribe) on Oct 03, 2007 at 10:47 UTC | |
by graff (Chancellor) on Oct 03, 2007 at 13:20 UTC |