You mean something like the following?

map { ref($_) eq 'SCALAR' ? $$_ : $_ } @$progress_output
[download]

Whilst I have always proven to be too lazy to create an account on PerlMonks, I have been a long time lurker.
Yet again a post from ikegami has helped me understand concepts I previously hadn't encountered or fully grasped the subtleties of.
I had seen the map function but had not grasped how I could utilise it and I have also just been educated regarding ternary operators.
Thank you ikegami for your helpful and educational post, the code works perfectly.

If I may impose on your time a little more, are there alternative algorithms that may achieve the desired end result? Is this the method you would use to achieve the outlined result or is there a better way?
I know better is rather subjective so to define better my interest lies in for improvements in regards to good programming practices, efficiency of code and also clarity of code.

I am not sure if this fits into what you're trying to do, but why not simplify the matter by building the output where and when you need it:

$ perl -e '#!/usr/bin/perl
> use warnings;
> use strict;
>
> my $count = 42;
> my $completed = 6;
>
> notify_progress();
> $completed = 7;
> notify_progress();
>
> sub notify_progress {
>     my $progress_output = ["\rProcessing: ", $count, " files to proc
+ess ; ", $completed, " files completed\n"];
>     print @$progress_output;
> }'
Processing: 42 files to process ; 6 files completed
Processing: 42 files to process ; 7 files completed
[download]

$ perl -e '#!/usr/bin/perl
> use warnings;
> use strict;
>
> my $count = 42;
> my $completed = 6;
>
> notify_progress();
> $completed = 7;
> notify_progress();
>
> sub notify_progress {
>     print "Processing: ",  $count, " files to process ; ", $complete
+d, " files completed\n";
> }'
Processing: 42 files to process ; 6 files completed
Processing: 42 files to process ; 7 files completed
[download]

Hi Laurent_R

Thanks for the feedback, I had considered building the entire output within the subroutine, however I have a requirement be able to modify the output based on the command line option chosen, for example on one run I may need the progress indication to be

Processing: 42 files to process ; 6 files completed
[download]

then on the next run the verbose switch may be selected so I might need

Processing: 42 files to process ; 2 files excluded ; 6 files completed
[download]

Using the array method I am currently looking at lets me just push the required reporting output into the array as I am processing the CLI options which seemed to me to be more efficient than reprocessing the CLI options every time I need to update the progress indicator.

Regardless I value the time you have taken to review and make suggestions, Thankyou.

You are welcome.

OK, now I understand why you're using an arrayref with references to the counters. Although I guess a simple array with references to the counters would also work.

An alternative approach would be to build dynamically the output subroutine depending on your input:

#!/usr/bin/perl
use warnings;
use strict;

my $verbose = shift // 0;
my $excluded = 2;
my $count = 42;
my $completed = 6;
my $notify_progress = build_output_subroutine(); 

$notify_progress->();
$completed = 7;
$notify_progress->();

sub build_output_subroutine {
    my @out;
    if ($verbose) {
        return sub  { print "Processing: ",  $count, " files to proces
+s ; ", $excluded, " files excluded; ", $completed, " files completed\
+n" };
    } else {
        return sub { print "Processing: ",  $count, " files to process
+ ; ", $completed, " files completed\n" };
    }
}
[download]

But, in this context, this probably does not really make things really simpler.

"I am looking for any suggestions ... that might lead to a more elegant solution."

One of the benefits of using lexical variables is that you can control their scope. Declaring all your variables at the start of runtime, and giving them file-scope, defeats this. You should use the smallest scope possible. You should generally also pass variables to your subroutines: there are exceptions to this such as in the code below; although, do note the anonymous block limiting the scope of the variables not passed in the function call (see also state for an alternative way to handle this).

Using an arrayref, as you've shown, is not something I would do. Using one of the string formatting functions (sprintf in the code below) would be a better choice. Adding a newline to each line of output, actually creates something more like a log than a progress indicator. Starting each output line with a carriage return (\r) is not doing what I think you hoped it would; the usual way to handle this is to use backspaces (\b) and then write the new indicator lines, effectively overwriting the previous indicator line.

Also note that writing lines without a terminal newline can cause buffering problems. I've set the special variable "$|" to overcome this; and localised that change so that it doesn't affect other I/O.

Here's example code implementing those techniques.

#!/usr/bin/env perl

use strict;
use warnings;

my @files = map { "file_$_" } 5 .. 14;

for (0 .. $#files) {
    show_progress(\@files, $_);
    # Process $files->[$_] here. For testing:
    sleep 1;
}

show_done();

{
    my ($prog_fmt, $last_msg_len);

    BEGIN {
        $prog_fmt = 'Processing "%s" (%d of %d) ...';
        $last_msg_len = 0;
    }

    sub show_progress {
        my ($files, $index) = @_;

        local $| = 1;
        my $clear = "\b" x $last_msg_len;
        my $msg = sprintf $prog_fmt, $files->[$index], 1+$index, 0+@$f
+iles;
        $last_msg_len = length $msg;
        print $clear, $msg;
        return;
    }
}   

sub show_done {
    print "\nProcessing completed.\n";
    return;
}
[download]

The first line of output looks like this:

Processing "file_5" (1 of 10) ...
[download]

That line is continually overwritten. The final output looks like this:

Processing "file_14" (10 of 10) ...
Processing completed.
[download]

Hi Ken

Thanks for your help,
To clarify I generally do pass variables into my subroutines, the use of global's in the sample code was for demonstration purposes, as was the newline feed at the end of the output line, it was purely intended to demonstrate the output without having to introduce delays.
I have never really understood the uses of anonymous blocks of code, your explanation and code sample has helped to clarify why I might need to use that method in the future, and I will also check out state as you recommend.
I have traditionally turned on Autoflush globally when I need it (as indeed I do for the purposes of the progress indication), I notice you are scoping it as a local to the subroutine, is there an advantage to this method? or alternatively a disadvantage to using autoflush globally?
sprintf is also something I have not used much, but I can see how your code could be adapted to achieve the variable output I require without resorting to references within the array so I will also put more effort into understanding how it works.

Thankyou for your assistance, it has been educational :-)

"Thanks for your help"

You're very welcome.

"I have traditionally turned on Autoflush globally when I need it ..., I notice you are scoping it as a local to the subroutine, is there an advantage to this method? or alternatively a disadvantage to using autoflush globally?"

Perl has a number of special variables (see perlvar). Some of these are set in a certain context ($a - sort; $1 - regex; $@ - eval; and so on). Others have default values which are usually fine and don't needed changing; code is written expecting these default values; changing them globally can mean that some code does not function as intended or expected. Changing them with local, in a limited scope, means the change does not affect other code:

$ perl -E 'say $|; { local $| = 1; say $| } say $|'
0
1
0
[download]

If you look in "Variables related to filehandles", you'll see (a few paragraphs down):

"You should be very careful when modifying the default values of most special variables described in this document. In most cases you want to localize these variables before changing them, since if you don't, the change may affect other modules which rely on the default values of the special variables that you have changed. ..."

That continues on with good and bad examples. A discussion of scope follows that:

"Usually when a variable is localized you want to make sure that this change affects the shortest scope possible. ..."

There are more good and bad examples; one showing use of an anonymous block.

Further related information can be found in perlsub; in particular, the sections "Temporary Values via local()" (especially the "Localization of special variables" subsection), and "When to Still Use local()".

— Ken

It looks like what you actually want is a templating system, like Template Toolkit

Hi Eily

To clarify the Template Toolkit is a web centric tool?
I am not actually interfacing this perl code with any kind of web front end, it is purely a CLI operation, however I had considered using a similar method with a multidimensional hash instead of an array, but when I thought about the complexities of keeping the ordering correct compared to my current requirements for this script an array just seemed simpler.
Anyhow, you have reminded me that I should take another look at Text::Template perhaps I could achieve some similar results using that.

Thanks for your assistance.

To clarify the Template Toolkit is a web centric tool?

Actually, no. It is a perfectly generic templating system - flexible and extensible. The breadth of the facilities which it offers has been recognised as a benefit by many web application developers and so they have used it for that purpose but it is not web-centric.

Here is your script re-written using TT2, for example:

#!/usr/bin/env perl 
use strict;
use warnings;
use Template;

my $t = {
    count     => 42,
    completed => 6
};
my $outtt2 =
  "\rProcessing: [% count %] files to process ; [% completed %] file c
+ompleted\n";
my $template = Template->new;

notify_progress ();

$t->{completed} = 7;

notify_progress ();

sub notify_progress {
    $template->process (\$outtt2, $t);
}
[download]

I've kept the notify_progress() subroutine so you can see the difference. In a real script this subroutine would disappear and just be replaced by the call to process() inline. If you have a need for templating (and most of us do sooner or later) then do take a look at TT2. It is big and has a steep learning curve compared to others but there are tutorials to help you through and if you use it as I have you will be far less likely to hit a limitation of your templating system than with other, lighter alternatives.

I agree with advice that a template is a good idea, even with a command line tool, but there is already a nice progress package specifically for command line tools: Time::Progress.

#!/usr/bin/env perl
use strictures;
use Time::Progress;
use Time::HiRes "usleep"; # <- just for demo.

my $total = 42;
my $progress = Time::Progress->new();
$progress->attr( min => 1, max => $total );

$| = 1;
for my $done ( 1 .. $total )
{
    usleep( rand 500_000 );
    print $progress->report("\rProcessing: $total files to process %L 
+%40B%E %p", $done);
}
print "\nDone!\n";
[download]

Update: fixed code booger; s/42/\$total/ in for loop.

Well, I can say I learned something today... I had no idea that you could separate integers with underscores to make it easier at a glance how long the number actually is.

perl -E 'say 500_000'
500000
[download]

Cool!

You can use underscores with decimal, hexadecimal, octal and binary numbers. You can also place them wherever you want, for convenient viewing of the data in question. They don't have to be a direct replacement of commas: both 50_00_00 and 500_000 are valid representations of 500,000; although, the first one is probably a poor choice because it's likely to be confusing.

Here's some highly contrived examples (purely to demonstrate those points):

$ perl -E 'say 1_6'
16
$ perl -E 'say 0x1_0'
16
$ perl -E 'say 02_0'
16
$ perl -E 'say 0b1_00_00'
16
[download]

See "perldata: Scalar value constructors" for details.

— Ken

Yes, I'm also a fan of Perl allowing underscores in numeric literals.

BTW, I was pleasantly surprised to learn that the C++ committee are also fans of this splendid idea:

• C++11 user-defined literals: These are extensible and use a trailing underscore. For example: 12_km
• C++14 digit separators: "The single-quote character ' can now be used anywhere within a numeric literal for aesthetic readability". For example: 500'000 or 500'000_km or pi = 3.14159'26535'89793. Presumably, single quote was chosen for the digit separator because underscore had already been taken by user-defined literals.

In fact, it's a perlism I am extremely fond of. I can easily see someone in a design discussion saying, What? We don't need that informal, random parsing trouble. It's just numbers. If you can't read numbers, you shouldn't be programming. That underscore has saved me many hours of reading comprehension and probably a hundred hours of chasing bugs and mistakes in code. Update: and scratches on my monitor as I try to count digits with pen or ruler. :P

Thanks for the advice about Time::Progress, That module has proven to be very useful.