Re^3: Split web page, first 30 lines only -- :content

Hello wrkrbeee,

I think Discipulus provided this sample code to demonstrate a useful approach which you can adapt to your particular needs. If you want to process the read-in lines in the calling code (your “Primary loop”) rather than in the callback function, then you need to store the lines in a shared variable rather than print them in sub head_only. There is an additional complication: the last line read from the current chunk of data may not be complete, so you need to check for a trailing newline and handle its absence appropriately:

use strict;
use warnings;
use LWP::UserAgent;
use feature 'state';
use constant LINES => 30;

my @pages = ('http://www.perlmonks.org', 'http://perldoc.org');
my $ua = LWP::UserAgent->new;
my $read_lines = 0;
my @lines;

for my $url (@pages)
{
    my  $response = $ua->get($url, ':content_cb' => \&head_only);

    if ($response->is_success)
    {
        print "$_\n" for @lines;
        print +("=" x 70), "\n";
    }
    else
    {
        die $response->status_line;
    }

    $read_lines = 0;
    @lines = ();
}

sub head_only
{
    state $last_line_incomplete  = 0;
    my ($data, $resp, $protocol) = @_;
    my  @all_lines = split "\n", $data;

    warn "LINES: ", scalar @all_lines, "\n";

    if ($last_line_incomplete)
    {
        my $first_line = shift @all_lines;
        $lines[-1] .= $first_line;
    }

    for my $line (@all_lines)
    {
        if (++$read_lines > LINES)
        {
            $last_line_incomplete = 0;
            die;
        }

        push @lines, "line $read_lines: $line";
    }

    $last_line_incomplete = $data !~ /\n$/;
}
[download]

print +("=" x 70), "\n"; #what is this?
[download]

The x operator creates a string of 70 equals characters concatenated together:

======================================================================
[download]

— see perlop#Multiplicative-Operators. The plus sign is there to prevent the Perl parser from thinking that the parentheses contain the entire argument list to the print function — see print.

Hope that helps,

Athanasius <°(((>< contra mundum Iustus alius egestas vitae, eros Piratica,

Comment on Re^3: Split web page, first 30 lines only -- :content_cb trick Select or Download Code

Replies are listed 'Best First'.
Re^4: Split web page, first 30 lines only -- :content_cb trick by wrkrbeee (Scribe) on Mar 01, 2017 at 14:41 UTC
Thank you Athanasius! Appreciate your time and effort!	[reply]