Large table query code help!

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi there Monks!
I am querying a large table 50 records at a time, that works no problem, but at line 34 this code:

$allaccounts = $sql_dbh->exec( "select * from othertable where acc in 
+($placeholders)", @files );
[download]

Is where I am having the issue I can't figure it out.
It seems that the account numbers in this variable "@files" has no comma separated between its values, this might be the cause, I am just confused on how to get it fixed. The reason might be because on line 30, if I use “push” adding those values into “@files, it will find them in the "othertable" table as you can see in this sample code I am posting here. Does the values in @files on line 34 needed to somehow have comma separated in order to the code to work? Any help its appreciated!
Here is the sample code for this:

#!/usr/bin/perl -w

use strict;
use CGI qw/:standard/;
use POSIX qw(strftime);

my $db = "test";
my $sql_dbh = lib::Db->connect( 'local' );

# get a list of account numbers
my @data = map { $_->{ ACCOUNT } } @{ $sql_dbh->exec( qq|select accoun
+t from my_table where type='A'|) };

if (!@data) {
  die "bye now";exit;
 }

my $numb = @data;
print "\n Numbers of Accounts: $numb\n";

# Check 50 records at time 
my $allaccounts;
for( my $i = 0; $i < @data; $i += 50 ) {

  my @files = @data[$i .. $i+49];

  #my $all = join(', ', @files);
  #print "\n $i - $all\n";

  #test data
  #push(@files, "11223","5776","0001","87100");    

  my $placeholders = join ',', ('?') x @files;

  $allaccounts = $sql_dbh->exec( "select * from othertable where acc i
+n ($placeholders)", @files );
 
  
      
}

my $cc=0;
foreach (@$allaccounts) {
  $cc++;
  # print report
  my $found = qq|$_->{ name }$_->{ number }|;
  print "\n $cc - Reporting: $found\n"; 


 }
[download]

Comment on Large table query code help! Select or Download Code

Replies are listed 'Best First'.
Re: Large table query code help! by moritz (Cardinal) on Feb 02, 2012 at 20:03 UTC
Where does the `lib::Db` come from? it doesn't seem to be `use`d anywhere. Anyway, you speak of "issues", but don't mention what the problem is. Is the database engine throwing an error? If yes, which? Or are you getting a different result than expected? If yes, what do you get, what do you expect? If you are unsure what values some variables contain, use Data::Dumper to find out. One problem I can see with your code is that if `@data` doesn't contain a number of items that is divisible by 50, then the last query will have lots of trailing `undef`s. Not sure if you want that. Perl 6 - second systems done right	[reply] [d/l] [select]
Re^2: Large table query code help! by Anonymous Monk on Feb 02, 2012 at 20:09 UTC
the last query will have lots of trailing undefs. Not sure if you want that. If the database library maps those to NULLs, it should not be a problem (they match nothing).	[reply]
Re: Large table query code help! by Anonymous Monk on Feb 02, 2012 at 20:05 UTC
What are you trying to achieve in your code? If you want account numbers that are both in table 1 and table 2, it can be easily done with SQL only. In table 1 but not in table 2? No problem. Database engines have been optimising for these sorts of things for decades, so it will very likely work faster and look cleaner than your current code. As I understand it, you query for the second table in batches of fifty, leave the last batch of (up to) 50 accounts in `$allaccounts`, and then print out that variable. Does the values in @files on line 34 needed to somehow have comma separated in order to the code to work? I am not familiar with your database library (nor do you anywhere mention which one you are using), but if it supports placeholders such as those you are using, I very much doubt that.	[reply] [d/l]
Re^2: Large table query code help! by Anonymous Monk on Feb 02, 2012 at 20:48 UTC
I see your point and as far as this code sample is, the issue here is that this line: `for( my $i = 0; $i < @data; $i += 50 ) { ...` should end after the last foreach loop, no db calls or whatever else should matter since you mentioned that all has been working, or: #!/usr/bin/perl -w use strict; use CGI qw/:standard/; use POSIX qw(strftime); my $db = "test"; my $sql_dbh = lib::Db->connect( 'local' ); # get a list of account numbers my @data = map { $_->{ ACCOUNT } } @{ $sql_dbh->exec( qq\|select accoun +t from my_table where type='A'\|) }; if (!@data) { die "bye now";exit; } my $numb = @data; print "\n Numbers of Accounts: $numb\n"; # Check 50 records at time my $allaccounts; for( my $i = 0; $i < @data; $i += 50 ) { my @files = @data[$i .. $i+49]; #my $all = join(', ', @files); #print "\n $i - $all\n"; #test data #push(@files, "11223","5776","0001","87100"); my $placeholders = join ',', ('?') x @files; $allaccounts = $sql_dbh->exec( "select * from othertable where acc i +n ($placeholders)", @files ); #} # not here my $cc=0; foreach (@$allaccounts) { $cc++; # print report my $found = qq\|$_->{ name }$_->{ number }\|; print "\n $cc - Reporting: $found\n"; } } # end here [download] Enjoy it !!!	[reply] [d/l] [select]