Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks!

I am working on this code that grabs lots of records from two big tables in two different databases, at some point I had to loop two connections to get the values back, the code is hanging and I think its on this part of my code, I might be missing something here that someone could see and tell me that it would run better and faster if it was done another way.
Can someone look over the code and try to tell me if Perl wise it would be a problem processing all the records based on this code?
Here is the code:

my $ibm; # Get today's date -1, no weekends::: #my $d_date = today() - 1; #$d_date-- while ($d_date->day_of_week == 0 or $d_date->day_of_week == + 6); # get the newly inserted data from today_records and calculate current + stuff using data from IBMDB::: my ($sql_net,$n_year, @new_date); my $db = "LocalServer"; my $user = $Mydb::Config::odbc_def->{ $db } { user }; my $pass = $Mydb::Config::odbc_def->{ $db } { pass }; my $dbh = DBI->connect("DBI:ODBC:$db",$user, $pass, {RaiseError => 1}) +; my $sql_last ="SELECT account_num, due_year, account_due_date, net FRO +M today_records WHERE type='OK'"; my $sth = $dbh->prepare($sql_last); $sth->execute() || die $sth->errstr; #$sth->finish; #Executes web DB #Going back to the IBMDB to process the rest of accounts from last yea +r:::: my $user = $Mydb::Config::odbc_def->{ $ibm }{ user }; my $pass = $Mydb::Config::odbc_def->{ $ibm }{ pass }; my $dbh = DBI->connect("DBI:ODBC:$ibm",$user, $pass) || print "Connect + fail: $!"; #New connection to update today_records with new data::: my $db = "LocalServer"; my $user = $Mydb::Config::odbc_def->{ $db } { user }; my $pass = $Mydb::Config::odbc_def->{ $db } { pass }; my $dbh_2 = DBI->connect("DBI:ODBC:$db",$user, $pass, {RaiseError => 1 +}); my $count=0; while (my $pointer = $sth->fetchrow_hashref){ my $account_num = $pointer->{'account_num'}; my $account_due_date = $pointer->{'account_due_date'}; my $net = $pointer->{'net'}; my $due_year = $pointer->{'due_year'}; #Format DATE and subtract 1 year from current year to get current ter +m here... if($account_due_date=~/(\d{1,4})-(\d{1,2})-(\d{2})(\s*.*)/g){ #new year $n_year = $year-1; #$n_year = "2006"; $account_due_date=$n_year."-".$2."-".$3; } #Format DATE if($due_year=~/(\d{1,4})-(\d{1,2})-(\d{2})(\s*.*)/g){ $due_year=$1; } #Get all accounts for current term::: my $sql_back="SELECT ABC1,ABC2,ABC3,ABC4,ABC5,ABC6, ABC7, ABC8 FROM TBIBM WHERE ABC2='$account_num' AND ABC6 = '$accou +nt_due_date' AND ABC5='$n_year'"; my $sth = $dbh->prepare($sql_back); $sth->execute() || die $sth->errstr; while (my $pointer = $sth->fetchrow_hashref){ $count++; $ABC1 = $pointer->{'ABC1'}; $ABC2 = $pointer->{'ABC2'}; $ABC3 = $pointer->{'ABC3'}; $ABC4 = $pointer->{'ABC4'}; $ABC5 = $pointer->{'ABC5'}; $ABC6 = $pointer->{'ABC6'}; $ABC7 = $pointer->{'ABC7'}; $ABC8 = $pointer->{'ABC8'}; #done getting all accounts , need to update web db once again w +ith new data! $sql_net="UPDATE today_records SET current_prem= CONVERT(money, '$ABC7') WHERE account_num='$ABC2' AND type='OK'"; my $sth = $dbh_2->prepare($sql_net); $sth->execute() || die $sth->errstr; } }

If any questions let me know!
Thanks a lot!!

Replies are listed 'Best First'.
Re: Perl Code Efficiency Issue
by Fletch (Bishop) on Feb 15, 2008 at 15:01 UTC

    One thing that jumps out at me is that you're constantly re-prepare-ing your UPDATE statement inside the last while loop. You should prepare it once outside the loop with placeholders (instead of interpolating into the string; in fact you should do that with your other statements as well), then just call execute with the values you've extracted out.

    Another performance hit might be that you're using fetchrow_hashref; I want to say using this rather than (say) fetchrow_arrayref incurs a higher overhead (since it's got to parse and populate a hashref rather than just returning a set of values).

    The cake is a lie.
    The cake is a lie.
    The cake is a lie.

      Can you give an example on how I would do that?
        Before the loop:
        $sql_net="UPDATE today_records SET current_prem= ? WHERE account_num= ? AND type='OK'"; my $sth2 = $dbh_2->prepare($sql_net);
        In the loop:
        $sth2->execute(CONVERT(money, '$ABC7'),'$ABC2') || die $sth2-> +errstr;
        Update: Changed name of $sth, since previous version is still in scope.

        If you want to be truly lazy, simply replace prepare by prepare_cached, and DBI itself will check if it has ever seen the query before, and will return an appropriate context for your statement handler (which in turn should hit a query plan cached by the DB backend).

        • another intruder with the mooring in the heart of the Perl

Re: Perl Code Efficiency Issue
by kyle (Abbot) on Feb 15, 2008 at 15:13 UTC

    I don't know about efficiency, but this is pretty confusing:

    my $sth = $dbh->prepare($sql_last); # ... while (my $pointer = $sth->fetchrow_hashref){ my $sql_back = '...'; my $sth = $dbh->prepare($sql_back); while (my $pointer = $sth->fetchrow_hashref){ # ... my $sth = $dbh_2->prepare($sql_net); # ...

    The rules of lexical scoping make this all work, but I highly recommend that each $sth and $pointer (which would be more accurately, but no more descriptively, called $hash_reference) have their own name. If you use warnings (and I recommend that too), these things will toss out messages like "my" variable $x masks earlier declaration in same scope.

    Does this actually hang, or just take a long time? Adding some print statements in your loops might help you see what it's doing. For general efficiency tuning, I recommend Profiling your code.

Re: Perl Code Efficiency Issue
by cdarke (Prior) on Feb 15, 2008 at 15:09 UTC
    The thing that hits me is the two connects to $dbh.
    my $dbh = DBI->connect("DBI:ODBC:$db",$user, $pass, {RaiseError => 1}) +; my $dbh = DBI->connect("DBI:ODBC:$ibm",$user, $pass) || print "Connect + fail: $!";
    You are executing a SELECT after the first one, but not getting the results. The $sth->finish is commented out, but that may give you a memory leek. You should probably disconnect the dbh before opening it again. I can't say that this will cause a "hang", but it doesn't look right.
Re: Perl Code Efficiency Issue
by Jenda (Abbot) on Feb 15, 2008 at 18:18 UTC
    SELECT ABC1,ABC2,ABC3,ABC4,ABC5,ABC6, ABC7, ABC8 FROM TBIBM WHERE ABC2='$account_num' AND ABC6 = '$account_due_date' AND ABC5='$n +_year'
    ???????
    • Step one, kill whoever designed the database schema.
    • Step two, hide the body, destroy any evidence.
    • Step three, create a sane schema, copy the data if necessary and dump the old database.

    Also if you have the whole row nicely stacked in a hash, don't bother copying it to separate variables.