Any framework for bulk operations?

ganeshk has asked for the wisdom of the Perl Monks concerning the following question:

I am not sure whether this is even a perl question. There would be lot of cases where you want to do a couple of operations on a set of objects( say n objects). Some of the operations may be time consuming and may cause huge performance problems when they are done n times. In order to optimize the code we may rewrite it such that we can loop through the objects and then do some operations once on all objects at once shot instead of doing them n times. This could make it faster. For eg. let's say when we want to generate a lot of rows in a report. We call a method report_row on all objects


sub report_row {
    my $object = shift;
    $object->op1();
    $object->op2(); # op2 is very slow
    $object->op3();
}

report_row($_) for ( @objects );
[download]

For optimizing, you may rewrite the code in such a way that


$_->op1() for ( @objects );

# bulk_op2 does the same job as op2 but with all objects in
# one shot
bulk_op2(@objects); 

$_->op3() for ( @objects );
[download]

An example where this situation can arise is suppose you do some database query to get some data using some object attribute. Doing it on all objects would require n query executions. You may convert the query to support n attributes and then execute the query in one shot for all objects. But I feel that the initial code is much more readable than the latter. Now is there any framework or pattern using which we can still have the code as the first kind(or atleast close to the first) and still make it work like the second. Is it possible to express in any way that some operation can be done on bulk?

Comment on Any framework for bulk operations? Select or Download Code

Replies are listed 'Best First'.
Re: Any framework for bulk operations? by dragonchild (Archbishop) on May 16, 2008 at 17:11 UTC
For databases, both DBIx::Class and Rose::DB support this sort of things in various ways. But, it's not expressed in this fashion, so you'll want to read through all the documentation to find it. Look for words like "prefetch". My criteria for good software: Does it work? Can someone else come in, make a change, and be reasonably certain no bugs were introduced?	[reply]
Re^2: Any framework for bulk operations? by ganeshk (Monk) on May 16, 2008 at 17:59 UTC
Hi Dragonchild, Can you please explain me how DBIx::Class or Rose::DB supports bulk database oprations? Thank you	[reply]
Re^3: Any framework for bulk operations? by dragonchild (Archbishop) on May 16, 2008 at 18:11 UTC
Let's say that you have a row with a number of child rows in some other table. You might not be interested in the child rows at all, in which case it's a waste to retrieve them at all be interested in only one, in which case it should be retrieved when you want it be interested in all of them, so it's best to retrieve them all immediately. Both DBIC and RoseDB support the ability for the user to specify how they want to handle those kinds of details. And, this is just one example. You'll need to read the documentation to really get an idea of what's possible. My criteria for good software: Does it work? Can someone else come in, make a change, and be reasonably certain no bugs were introduced?	[reply]
Re: Any framework for bulk operations? by BrowserUk (Patriarch) on May 16, 2008 at 18:18 UTC
There is no way to predict how many (or which) other objects you will be calling `->op2()` on, when you call `report_row()` on the first of them. Or the last for that matter. However, if you wrap up your collection (array) of objects in a class, then you can provide a `->report_rows()` method that knows how many (and which) objects you intend to process and so it can aggregate the individual steps if that makes sense. Of course, when I say "it can aggregate", that really means "you can aggregate" when you write that method. (Of course, you could write `report_rows()` as a function equally well.) There isn't, and probably could not be, any mechanism that could analyse your `report_row()` function and automatically generate a `report_rows()` function from it. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. "Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."	[reply] [d/l] [select]