in reply to Counting fields from database

I need to know the total number of threadIDs

It's a little unclear to me from your description whether you just need to know this information, in which case you can ask the database to provide it with a GROUP BY clause, or whether you're looping through all messages anyway, and you just want to count thread IDs*. In the latter case, since I assume the thread IDs are a unique identifier per thread, you can do this with a hash. Here are examples of both:

use warnings;
use strict;
use Data::Dump;
use DBI;

# set up a dummy database
my $dbh = DBI->connect("dbi:SQLite::memory:", undef, undef,
    { RaiseError=>1 });
$dbh->do(<<'END');
CREATE TABLE messages (
    msgID INTEGER,
    threadID INTEGER
)
END
my $sth_i = $dbh->prepare(
    'INSERT INTO messages (msgID, threadID) VALUES (?,?)');
$sth_i->execute($_,$_>>4) for 0..127;

# ask the DB to give us per-thread information
dd $dbh->selectall_arrayref(
    'SELECT threadID, COUNT(*) FROM messages GROUP BY threadID');
# => [ [0, 16], [1, 16], [2, 16], [3, 16],
#      [4, 16], [5, 16], [6, 16], [7, 16], ]

# loop through rows and count threads ourselves
my $sth_s = $dbh->prepare(
    'SELECT msgID, threadID from messages');
$sth_s->execute;
my %threadIDs;
while ( my $row = $sth_s->fetchrow_hashref ) {
    $threadIDs{ $row->{threadID} }++;
}
dd \%threadIDs;
# => { "0" => 16, "1" => 16, "2" => 16, "3" => 16,
#      "4" => 16, "5" => 16, "6" => 16, "7" => 16 }

[download]

* Update: Upon rereading, it's probably the latter. By the way, $current_count_all <= $stopcount_all sounds like you want to limit the number of records returned, which you can also do in the database, e.g. in MySQL with a LIMIT clause.

Update 2: For completeness, there are two issues with the approach you showed in the OP. First, you need to be certain that $threadID doesn't ever contain the separator character ("," in this case), and your regex is too simple in that it will also match partial IDs (e.g. /234/ will match in "1234,5678"), which you'd need to prevent by anchoring the regex appropriately - for example, if the IDs are integers, you could say /\b\Q$threadID\E\b/) (note I've used \Q...\E to escape any special characters in $threadID even though I just said they're integers; it's just to play it extra safe). But the method of searching the string for the $threadID will be way less efficient than a hash, so I would always recommend that instead. Also modified the first example to supply the threadID in addition to the count.

Replies are listed 'Best First'.
Re^2: Counting fields from database (updated)
by htmanning (Friar) on Oct 03, 2020 at 00:49 UTC
    Thanks for this. I've updated my regex with what you suggested. You're right, mine was too simple.

    I used this example from someone else below and it works, but I still have to query the database twice. I can't seem to figure out how to do it in one loop, but it works. I loop through to count the total threads, then loop through again to print the rows.

[reply]
Re^2: Counting fields from database (updated)
by htmanning (Friar) on Oct 03, 2020 at 01:31 UTC
    I think my logic is flawed. This is a better example of what I'm doing:

    This is the query of the database: 

    Select * from messages where (username='$username' or toname='$usernam
+e') and message!='' order by dateadded desc, ID desc

    [download]
    This follows: 
            $titlesperpage = 10;
        $currentpage = 1;
        $startcount = ($currentpage - 1) * $titlesperpage + 1;
        $stopcount = $currentpage * $titlesperpage;
        $current_count = 0;
        $threadcount=0;
        $threadIDlist = "";
         while ($pointer3 = $sth3->fetchrow_hashref){
            $current_count++;
            $threadID = $pointer3->{'threadID'};
            $threads{$pointer3->{'threadID'}}++; 
            
            
                if ($threadIDlist !~ /$threadID/) {
                    $threadIDlist=$threadID . "," . $threadIDlist;
                    $current_count--;
                    $threadcount++;    
                 }
                 
           }

    [download]
    I try to only print 10 records per page. It works perfectly for the first page, but when I come back to print the second page it duplicates some threadIDs because they are not yet in the $threadIDlist var. The second page is called with this query_string: 
    ?currentpage=2

    [download]
[reply]
[d/l]
[select]

      Sorry, but your code is not runnable, and doesn't seem to be representative of the problem you're having. Please see How do I post a question effectively?, I know what I mean. Why don't you?, and Short, Self-Contained, Correct Example.

      This is a guess of what you want that I've pieced together from your various posts.

      use warnings;
use strict;
use Data::Dump;
use DBI;

my $titlesperpage = 10;

# set up a dummy database
my $dbh = DBI->connect("dbi:SQLite::memory:", undef, undef,
    { RaiseError=>1 });
$dbh->do(<<'END');
CREATE TABLE messages (
    msgID INTEGER,
    threadID INTEGER
)
END
my $sth_i = $dbh->prepare(
    'INSERT INTO messages (msgID, threadID) VALUES (?,?)');
$sth_i->execute($_,$_>>4) for 0..127;

for my $currentpage (1..5) { # simulate requests for different pages
    
    my $startcount = ($currentpage - 1) * $titlesperpage + 1;
    my $stopcount = $currentpage * $titlesperpage;
    dd $startcount, $stopcount;
    
    my $sth_s = $dbh->prepare('SELECT msgID, threadID FROM messages');
    $sth_s->execute;
    my %threadIDs;
    my $rowcnt = 1;
    while ( my $row = $sth_s->fetchrow_hashref ) {
        $threadIDs{ $row->{threadID} }++;
        if ( $rowcnt >= $startcount && $rowcnt <= $stopcount ) {
            dd $row->{msgID};
        }
    } continue { $rowcnt++ }
    dd \%threadIDs;
    
}

      [download]

      But this loops through all results for every request. You can have the database do the pagination, even though you have to hit the database twice it still doesn't require you to loop through all records, only those for the current page.

      for my $currentpage (1..5) { # simulate requests for different pages
    
    my $startcount = ($currentpage - 1) * $titlesperpage + 1;
    
    dd $dbh->selectall_arrayref(
        'SELECT threadID, COUNT(*) FROM messages GROUP BY threadID');
    
    my $sth_s = $dbh->prepare('SELECT msgID, threadID FROM messages '
        .'ORDER BY msgID LIMIT ?,?');
    $sth_s->execute($startcount, $titlesperpage);
    while ( my $row = $sth_s->fetchrow_hashref ) {
        dd $row->{msgID};
    }
    
}

      [download]
[reply]
[d/l]
[select]
        Thanks for taking the time to help, but this is above my pay grade. I couldn't get this to work. I ran both and they return the pages without errors but also without the vars I need like the $ID, etc. I'm doing something wrong. As I have it now it works as long as it only displays page 1. As soon as I move to page 2 (which calls the same URL but with the added querystring ?$currentpage=2) it reruns the database search and includes threadIDs I already printed on p. 1. As I read the code it is simply poorly constructed. Like I said, it's above my pay grade.

        This script was written years ago in Perl 4 and I'm trying to go through and fix things but I need to just hire someone to do it. I think I will just increase the titlesperpage to 50 and call it a day. Nobody should get that many messages.


        UPDATE: I may have made some progress. If I use the GROUP BY threadID in the SQL search, I can get rid of this if statement: 

        if ($threadIDlist !~ /\b\Q$threadID\E\b/) {
}

        [download]
        And I don't think I need to count the total number of threads anymore. I've commented that section out below and it seems to work.

        I end up with the following. Please forgive me if this is not "runable" as your previous note suggests. The connections to the database are wrapped in subroutines. I've been told this is a bad way of doing it but that's what is in here. 

            
    
    $currentpage = param('currentpage');
    
    &Conn_to_DB;
    $SQL =  "Select * from messages where (username='$username' or ton
+ame='$username') and message!='' GROUP BY threadID order by dateadded
+ desc, ID desc";
    &DoSQL;
    $recordcount = $sth->rows;


    if ($recordcount <= 0) {
      print "Error message goes here.";
    } else {
    
    
# Separately counting the total number of threads is no longer needed 
+(apparently).    
#   $SQL2 = "Select * from messages where (username='$username' or ton
+ame='$username') and message!='' order by ID desc";
#     &DoSQL2;
#          %threads;
#          while ($pointer2 = $sth2->fetchrow_hashref){
#             $threads{$pointer2->{'threadID'}}++; 
#            }
#            $threadcount_all = scalar keys %threads; 

           
    
        $titlesperpage = 10 if ($titlesperpage eq "");
        $currentpage = 1 if ($currentpage eq "" || $currentpage < 1);
        $startcount = ($currentpage - 1) * $titlesperpage + 1;
        $stopcount = $currentpage * $titlesperpage;
        $current_count = 0;
        $threadcount=0;
        $threadIDlist = "";
        
        while (($pointer2 = $sth2->fetchrow_hashref) && ($current_coun
+t <= $stopcount)){
            $current_count++;
  
            if ($current_count >= $startcount && $current_count <= $st
+opcount) {

            $name = $pointer2->{'name'};
            $message = $pointer2->{'message'};
            $date = $pointer2->{'date'};
        
        
            #this is no longer needed    
            #if ($threadIDlist !~ /\b\Q$threadID\E\b/) {
            #$threadIDlist=$threadID . "," . $threadIDlist;
            #$threadcount++;    
        
        
        
        print qq~ Content goes here~;
    
    
    
            #} end if ($threadIDlist !~ /\b\Q$threadID\E\b/) {
        
          } #end if ($current_count >= $startcount && $current_count <
+= $stopcount) {
      
    } #end while (($pointer2 = $sth2->fetchrow_hashref) && ($current_c
+ount <= $stopcount)){
           
           
           
sub Conn_to_DB{
  use DBI;
  $DSN = "DBI:mysql:database_name:db.domain.com";
  $sqluser = "user";
  $sqlpass = "pw";
  $dbh = DBI->connect($DSN,$sqluser,$sqlpass)
    || die "Cannot connect: $DBI::errstr\n" unless $dbh;
  return;
} 

           
sub DoSQL{
  eval {
    $sth = $dbh->prepare($SQL);
  }; # end of eval

  # check for errors
  if($@){
    $dbh->disconnect;
    print "Content-type: text/html\n\n";
    print "An ERROR occurred! $@\n";
    exit;
  } else {
    $sth->execute;
  } # end of if/else
  return ($sth);
}


sub DoSQL2{
  eval {
    $sth2 = $dbh->prepare($SQL2);
  }; # end of eval

  # check for errors
  if($@){
    $dbh->disconnect;
    print "Content-type: text/html\n\n";
    print "An ERROR occurred! $@\n";
    exit;
  } else {
    $sth2->execute;
  } # end of if/else
  return ($sth2);
}

        [download]

        The above seems to work. It returns only the number of threads as titlesperpage asks for. Please let me know if I'm wrong.

        Really appreciate the help.

[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom