perlancar has asked for the wisdom of the Perl Monks concerning the following question:

tqdm is a Python progress bar/meter library which has a simple interface: you just wrap an iterable:

for i in range(100): ...

with:

for i in tqdm(range(100)): ...

and voila, you have a progress bar. I wonder how we can do something similar in Perl. If we create a tqdm() function which returns a tied array, e.g.:

for (tqdm(1..100)) { ... }

then Perl will FETCH all the items first before starting the loop, defeating the progress measuring.

We can make tqdm() function return a tied scalar for every element, but that will be more inefficient and introduce additional side effects.

We can have tqdm() return a tied scalar, something like:

for (my $i = tqdm(100); $i < 100; $i++) { ... }

but: 1) $i is still a tied scalar; 2) C-style loop is less nice and less often used; 3) the above syntax is repetitive.

Any ideas?

UPDATE 2019-04-16: We did it! In Perl we can indeed support the same syntax by using a combination of tied array and lvalue wrapper. Read all about it in Getting for() to accept a tied array in one statement.

Replies are listed 'Best First'.
Re: Porting tqdm to Perl (updated)
by haukex (Archbishop) on Dec 21, 2018 at 10:56 UTC

    The typical approach in Perl would be a while loop...

    my $it = tqdm(100); while (my $i = $it->next) { ... }

    Plenty of variations are possible, for example, $it doesn't need to be an object, it could just be a code ref; or, if it is an object, it can overload the <> operator (while(<$it>)), etc.

    BTW, Higher-Order Perl is great for this kind of stuff.

    Update: One advantage of overloading <> is that a while(<$it>) is automatically translated to while (defined($_ = readline $it)), meaning that only undef terminates the sequence, and not other false values like 0:

    use warnings; use strict; { package Iterator; use overload '<>' => sub { my $self = shift; if (wantarray) { my @all; while (defined( my $x = $self->() )) { push @all, $x } return @all; } else { return $self->() } }; sub new { my $class = shift; my @values = @_; my $i=0; return bless sub { if ($i>=@values) { $i=0; return } return $values[$i++]; }, $class; } } use Test::More tests=>2; my $it1 = Iterator->new(-5..5); my @o1; while (<$it1>) { push @o1, $_; } is_deeply \@o1, [-5,-4,-3,-2,-1,0,1,2,3,4,5]; SKIP: { skip "need Perl >= v5.18 for overloaded <> in list context", 1 unless $] ge '5.018'; # [perl #47119] my $it2 = Iterator->new(-5..5); is_deeply [<$it2>], [-5,-4,-3,-2,-1,0,1,2,3,4,5]; }

      Yes, creating an iterator (either a full object or just a coderef) would be one of the obvious choices. Overloading the diamond operator is interesting, thanks for pointing that out. Sadly, all these are still far from the "just add 4+2 characters to do progress bar" that the Python's tqdm libary enables.

      As someone wrote in How to Write Perfect Python Command-line Interfaces — Learn by Example: "Do you see any difference? It is not so easy to spot because the difference consists of 4 letters: TQDM. (...) This is the name of a Python library and this is the name of its unique class, with which you wrap any iterable to print the corresponding progression. (...) And this results in a beautiful progress bar. Personally, I still find it too good to be true."

      It's seldom that I feel Python envy, but this is one of those moments :-)

Re: Porting tqdm to Perl
by Corion (Patriarch) on Dec 21, 2018 at 15:46 UTC

    I wrote Progress::Indicator, which tries to infer the "progress" through a list or file by looking at the call location, the scalar value passed to it or whether the argument passed to it is a filehandle and the position within that filehandle.

    The usage is a bit more complicated than tqdm because you pass it the iterated item (or something acting as identifier), and a message describing what is dome:

    use Progress::Indicator qw'progress'; for my $image (@files) { progress( \@files, "Processing $capture_date" ); };

    The output will be something like:

    Processing 2018-12-21 (10 of 1000, 10%, Remaining 1:10:59)

    ... or when the total size of the iterated item cannot be identified

    Processing 2018-12-21 (10 of 1000, 3/s)

    Process::Indicator was spun out of App::Photoimport, which should explain its lack of documentation etc. - maybe I find time to add documentation to it. I didn't find it worth releasing on CPAN because there isn't a lack of progress indicators on CPAN IMO.

Re: Porting tqdm to Perl
by LanX (Saint) on Dec 21, 2018 at 13:54 UTC
    Maybe you should better explain what tqdm actually does, I have to guess in the following:

    IIRC are (should be?) all iterators in Python objects inheriting from a class Iteratable.

    tqdm can just take one such object and return another enhanced iteratable one. (please correct me if I'm wrong)

    Perl has a multitude of different iterators, finding a standard procedure to change them should be difficult.

    I think the best way to go is to write tqdm into the loop (if it does what I understand)

    for (1..100) { tqdm; # same as tqdm($_); ... }

    Now the problem you'd be facing is extra code for initialisation, i.e. how can you tell that the loop was freshly entered or not?

    IMHO the default answer in Perl is to initialize that tqdm right before the loop starts.

    for my $outer (1..10) { my $tqdm = ProgressBar->new(); # or another constructor for (1..100) { &$tqdm; ... } }

    Of course TIMTOWTDI, but I hope this helps.

    Cheers Rolf
    (addicted to the Perl Programming Language :)
    Wikisyntax for the Monastery FootballPerl is like chess, only without the dice

      In an answer to haukex, I've included a link to a blog post that mentions how tqdm might be used. The tqdm PyPI page itself (which I linked in the original post) is also very informative already. But as usual I probably didn't communicate clearly what exactly I want to accomplish, so let me reiterate: how would one create a progress bar library in Perl that is roughly as simple to use to an existing script. By simple I mean with as minimal change as possible, adding as few characters of code as possible. And with existing script, this implies that some uses for() and some uses while() (I personally use for() far more often than while()).

      In this interface:

      for (1..100) { tqdm; # same as tqdm($_); ... }

      we can just assume that the first time tqdm() is first invoked is the start of the process. But tqdm() didn't receive the target number (100).

        and how does tqdm handle infinite iterators?

        or how would you handle readline without knowing the number of lines?

        or circular iterators?

        > But as usual I probably didn't communicate clearly what exactly I want to accomplish

        indeed ;-)

        Cheers Rolf
        (addicted to the Perl Programming Language :)
        Wikisyntax for the Monastery FootballPerl is like chess, only without the dice

Re: Porting tqdm to Perl
by kcott (Archbishop) on Dec 22, 2018 at 09:15 UTC

    G'day perlancar,

    I'm not particularly familiar with Python, and have never heard of tqdm(), so this is just a suggestion.

    The Smart::Comments CPAN module achieves something similar to what I think you're looking for. Its "Progress Bars" section has substantial documentation; also see the "Time-Remaining Estimates" section.

    It also documents a number of caveats which may be pertinent depending on your intended usage. For instance, it uses source filters and has some efficiency issues when the number of iterations is indeterminate.

    — Ken

      Adding progress bar using Smart::Comments is cute, and has its pro's too, but is not a solution I prefer in real-world practice. Actually you don't need to know much Python to understand the library, but it helps to know that Python uses iterable a lot and that is advantage for tqdm.

Re: Porting tqdm to Perl
by LanX (Saint) on Dec 21, 2018 at 14:02 UTC
    Another way to go might have been to use an attribute for you iterator variable, which ties it into a ProgBar tie-scalar class at declaration time.

    Unfortunately using attributes in for loops doesn't seem to be implemented.

    DB<58> for my $i :progbar (1..10) { print $i } syntax error at (eval 69)[C:/Perl_524/lib/perl5db.pl:737] line 2, near + "$i :"

    and of course this wouldn't work with implicit $_.

    Cheers Rolf
    (addicted to the Perl Programming Language :)
    Wikisyntax for the Monastery FootballPerl is like chess, only without the dice