Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi there Monks!
I just cant see it anymore, but I have to execute a query with account numbers in this array that will have at least 20000 values in it, well I need to check in three other tables if they match and get rid of them after reporting on them. My process of not letting all the 20000 into the "IN Operator" on the SQL query works well for me. But my question here is why I am loosing the value of this array (@acc_list) after the first "while"?
Any suggestions would be nice, here is a test code to express my frustration:
#!/usr/bin/perl -w use strict; use CGI qw/:standard/; my @acc_list = (00001 .. 1000); # this will have at lest 20000 accout +numbers in if (!@acc_list) { die " Nothing at this time";exit; } # From 1 table my $c_1=0; my $tb1; while ( @acc_list ) { $c_1++; my @files1 = splice( @acc_list, 0, 100 ); print "\n23 - @files1\n"; #my $holders_one = join ',', ('?') x @files1; #$tb1 = $dbh->exec_select( qq|select * from table1 where acc in ($ho +lders_one)|,@files1 ); } foreach (@$tb1) { print "\n1- Found 1\n"; } print "\n\n 33 - @acc_list \n\n"; # From 2 table my $c_2=0; my $tb2; while ( @acc_list ) { $c_2++; my @files2 = splice( @acc_list, 0, 100 ); #my $holders_two = join ',', ('?') x @files2; #$tb2 = $dbh->exec_select( qq|select * from table2 where acc in ($ho +lders_two)|,@files2 ); } foreach (@$tb2) { print "\n 2- Found 2\n"; } print "\n\n 52 - @acc_list \n\n"; # From 3 table my $c_3=0; my $tb3; while ( @acc_list ) { $c_3++; my @files3 = splice( @acc_list, 0, 100 ); #my $holders_three = join ',', ('?') x @files3; #$tb3 = $dbh->exec_select( qq|select * from table3 where acc in ($ho +lders_three)|,@files3 ); } foreach (@$tb3) { print "\n 3- Found 3\n"; } print "\n\n68 - **@acc_list** \n\n"; print "\n\n Done \n\n";
Thanks for looking!

Replies are listed 'Best First'.
Re: Loosing value and my mind on this one, help!!!
by BrowserUk (Patriarch) on Dec 01, 2011 at 20:16 UTC
    why I am loosing the value of this array (@acc_list) after the first "while"?

    Because that is the way you have programmed the loop:

    while ( @acc_list ) { ... my @files1 = splice( @acc_list, 0, 100 ); ... }

    You've literally written, "remove (splice) the elements out of the array @acc_list in batches of 100 until the array is empty".

    If you want to retain the values in the array, don't use splice.

    You might use something like this instead:

    for( my $p = 0; $p < @acc_list; $p += 100 ) { ... my @files1 = @acc_list[ $p .. $p+99 ]; ... }

    There are several other ways to code it, but that is probably the clearest.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

    The start of some sanity?

      Thank you, I was blind, now I can see!
Re: Loosing value and my mind on this one, help!!!
by TJPride (Pilgrim) on Dec 02, 2011 at 09:32 UTC
    FYI, SQL queries do have a maximum length, and you may crap out your query if you specify too many tens of thousands of items inside the query. It's also n squared efficiency to run, since it has to compare every item specified against every account in each of the tables without using an index. Depending on how many items are in each of those tables, or how often this needs to run, there may be a much better way to do it.
      I know on the max length on SQL queries, but this code was not bad at all based on letting a 100 values into the IN Operator at a time( out of more than 20000 values in the array been used). I couldn't find anything to use with Perl and SQL that would be better than this, but if anyone could and would show me some code it would be nice too. Thanks very!!!
        Can you describe your table structure? Go into the command line and type SHOW CREATE tablename, where tablename is each of your tables one at a time. This, incidently, is an excellent way to get index names if you didn't name your indexes and need to remove one.