Class::DBI Pagination with Table Joins

jgallagher has asked for the wisdom of the Perl Monks concerning the following question:

I'm wondering how you guys have solved this problem before, because I haven't been able to find anything (directly) applicable on CPAN. I need to paginate the results from some SQL queries which may or may not just be simple CDBI methods. (If they aren't, I can always use set_sql to create one.)

Using the canonical Music/CD/Artist setup, essentially what I want to do is search for all CDs by a certain musician with the results split up by page. I've found two direct options, both mentioned by the CDBI Wiki:

Class::DBI::Pager is marked as obselete because it actually returns the entire dataset and chops it up in memory. While I don't anticipate an extremely large amount of data, I could see 10,000 or 20,000, so this is probably not the best solution.
Class::DBI::Plugin::Pager is the listed alternative. Unfortunately, it is based on SQL::Abstract, which, as far as I can tell, only works on single tables. Even a fairly simple join like the one I described above is impossible.

Is what I'm doing that unusual? Am I completely missing something? I wondered at one point if this is something I should be doing at the database level, but I'm not even sure what to search for if that's the case. It seems like this would be something that many have run across before, so how did you handle it?

Comment on Class::DBI Pagination with Table Joins Download Code

Replies are listed 'Best First'.
Re: Class::DBI Pagination with Table Joins by saberworks (Curate) on Oct 13, 2005 at 14:42 UTC
And here's the problem with trying to abstract stuff like this. Either you pull it all into memory and chop it up that way, or you try to chop up the SQL query and rewrite it to do your pagination for you. I recently started a new job and they have a layer that psuedo-parses the SQL and does pagination, but of course it's problematic when you have "fancy" queries. If your database supports subqueries you can wrap an external query around your regular query and do pagination that way. Better yet, if your database supports views, create a view for each of these and make it appear that they are single tables. That way, the dumb modules can figure things out.	[reply]
Re: Class::DBI Pagination with Table Joins by zby (Vicar) on Oct 13, 2005 at 14:56 UTC
You might look at Class::DBI::Sweet (look for the 'Automatic joins for search and count' line in the POD) or the experimental DBIx::Class. The second one is meant to be a replacement for Class::DBI with lots of nice features, but I feel it will take time until it will be usefull for production.	[reply]
Re^2: Class::DBI Pagination with Table Joins by jgallagher (Pilgrim) on Oct 13, 2005 at 15:18 UTC
Nice. I wasn't aware of Class::DBI::Sweet. Just glancing at it, I'm not completely convinced it will do everything I need, but it may. It's definitely closer to what I need than anything else I've seen. Thanks++.	[reply]
Re: Class::DBI Pagination with Table Joins by perrin (Chancellor) on Oct 13, 2005 at 16:20 UTC
I use this patch for it with MySQL.	[reply]
Re: Class::DBI Pagination with Table Joins by pboin (Deacon) on Oct 13, 2005 at 14:58 UTC
Why do you want to paginate the data? For human consumption? If so, then I say "Don't do that.". And here's why: Nobody can grok that much data in the first place. Use SQL to give them the records they need at that point in time. For example, start with the top/first CD, LIMIT (an SQL keyword) 30. Then, do an OFFSET (another SQL keyword) 30, LIMIT 30, and so on... Of course if this isn't for human consumption, well, then, nevermind... <g>	[reply]
Re^2: Class::DBI Pagination with Table Joins by thor (Priest) on Oct 13, 2005 at 15:08 UTC
I thought that pagination meant giving the data to a user a chunk at a time. Which is why your post confuses me; you say "don't do that" and then say "here's how to impliment pagination". Also, I don't think that LIMIT is as universal as you might think. It works in Postgres and MySQL, but I know that it doesn't work in Sybase at the time of this writing. thor `Feel the white light, the light within Be your own disciple, fan the sparks of will For all of us waiting, your kingdom will come`	[reply]
Re^3: Class::DBI Pagination with Table Joins by pboin (Deacon) on Oct 13, 2005 at 15:18 UTC
You're right, I wasn't clear, sorry. What I meant was: Yes, do the pagination, but No, don't pull X-thousand records to do it. Paginate by pulling only what you need. (Think Just-In-Time inventory, except w/ records.) I can't speak Sybase SQL, and we all know that sometimes SQL seems like it's more dialect than ANSI. I've used LIMIT and OFFSET with success in my own work though, and it's really handy for this kind of work.	[reply]
Re^4: Class::DBI Pagination with Table Joins by herveus (Prior) on Oct 13, 2005 at 18:05 UTC
Re^2: Class::DBI Pagination with Table Joins by jgallagher (Pilgrim) on Oct 13, 2005 at 15:13 UTC
Like thor, I'm curious as to what you mean by "paginate" - your second paragraph describes how I would go about doing it by hand. But, this being Perl, I'd rather not do anything by hand that has already been done by someone else. :-)	[reply]
Re: Class::DBI Pagination with Table Joins by snoopy (Curate) on Oct 14, 2005 at 00:26 UTC
The sql, using the offset and limit clauses can be used to do server-side filtering. cdbi's construct method can then be used to substantiate objects: `my $page_size = 24; my $page_num = 3; my $offset = ($page_num - 1) * $page_size; my $query = "SELECT items.* from orders JOIN items ON items.order_ +id = orders.id WHERE orders.status='ready' ORDER BY orders.date OFFSE +T $offset LIMIT $page_size"; my $sth = MyClass::db_Main->prepare( $query ); $sth->execute; my @items; # # Construct this page of item objects # while (my $item_data = $sth->fetchrow_hashref()) { push(@items, MyClass::Item->construct($item_data)); }` [download]	[reply] [d/l]