Using lazy list with a for loop

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I am processing data from a webservice and my code has many instances of this that processes different types of data:

my $data = $api->request(...);
for my $item ($api->collect($data)) {
    process($item);
    for my $widget ($api->gather($item->{sets}{widgets})) {
        process($widget);
    }
}
[download]

The collect method converts the parsed data into a list of items, and if a pagination indicator is found, it makes another api request, parses the data and adds it to the accumulator, ..., until no pagination indicator is found, then returns the accumulated items.

This is not a good design, since the memory usage can rise unacceptably high, so I'd like to change the implementation details of collect to make use of an iterator or tied array and avoid accumulating, but I would also like to keep the current user interface of simply iteratoring over the results with a for loop. Is this possible?

Comment on Using lazy list with a for loop Select or Download Code

Replies are listed 'Best First'.
Re: Using lazy list with a for loop (updated) by haukex (Archbishop) on Jan 02, 2023 at 23:18 UTC
... make use of an iterator or tied array and avoid accumulating, but I would also like to keep the current user interface of simply iteratoring over the results with a for loop. Is this possible? You could use a tied array in a `for` loop if the number of items is known in advance (`FETCHSIZE` is called; see also my reply to the relatively recent thread How can I create an iterator that works inside foreach()). Otherwise I recommend you bite the bullet and switch to `while`, as that's generally the way to work with iterators in Perl, and iterators are incredibly powerful, as HOP taught us. Such a `while` can often also be expressed as a C-style `for(;;)`, but IMHO that doesn't always look as elegant as `while` and `foreach`. Update: Another alternative is to provide a custom API for looping with a callback, for example like Mojo::Collection's `->each`. I think it would be less performant than a `while`, though.	[reply] [d/l] [select]
Re^2: Using lazy list with a for loop (updated) by Anonymous Monk on Jan 03, 2023 at 00:29 UTC
Thank you for the confirmation.	[reply]
Re: Using lazy list with a for loop by LanX (Saint) on Jan 02, 2023 at 23:44 UTC
> but I would also like to keep the current user interface of simply iteratoring over the results with a for loop. Is this possible? In order to help you better, we need to know why. Put differently: Why not a `while` loop? Cheers Rolf _{(addicted to the 𐍀𐌴𐍂𐌻 Programming Language :) Wikisyntax for the Monastery}	[reply]
Re^2: Using lazy list with a for loop by Anonymous Monk on Jan 02, 2023 at 23:58 UTC
Because it is the user interface. I don't mind changing the implementation details behind the scenes, but I'd like to avoid inconveniencing others if possible. It's also more code- a single for loop has to be replaced by 2 statements, 1 to create the iterator, and the while loop.	[reply]
Re^3: Using lazy list with a for loop by LanX (Saint) on Jan 03, 2023 at 00:15 UTC
Merging the semantics of `for` and `while` in Perl is very hard. I experimented for years and it always came with a penalty, like performance or fuzzy variable scope. E.g. you can use syntactic sugar to define 2 subs `from(&;@) loop(&)` with prototypes to define your own loop construct like `from { LIST \| ITER } $a,$b => loop { BODY }` an `from` would check if ITER was blessed to an iterator class and act accordingly. But you can guess that this would be slow. I suppose even if implemented in XS it would be slow. BUT "it is ONE user interface". Are you willing to pay that price? Cheers Rolf _{(addicted to the 𐍀𐌴𐍂𐌻 Programming Language :) Wikisyntax for the Monastery}	[reply] [d/l] [select]
Re^4: Using lazy list with a for loop by Anonymous Monk on Jan 03, 2023 at 00:26 UTC
Re^5: Using lazy list with a for loop by LanX (Saint) on Jan 03, 2023 at 00:33 UTC
Some notes below your chosen depth have not been shown here
Re: Using lazy list with a for loop by Anonymous Monk on Jan 02, 2023 at 22:38 UTC
* s/->gather/->collect/	[reply]
Re^2: Using lazy list with a for loop by erzuuli (Cannon) on Jan 03, 2023 at 03:54 UTC
If you log in, you can fix your own nodes.	[reply]