Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks,
My question is regarding having a large 100MB text file, if I want to open this file and have an option to a user to only show 100 lines at the time, how could I accomplish such a thing? Stuck here
#=comment open VIEWFILE, "< $path" or die "Can't open $patht: $!<br>\n"; while( my $v_file = <VIEWFILE>) { for (my $count=0; <VIEWFILE>; $count++) { print "<div STYLE=\"font-family: 8pt Courier, 'Courier + New', monospace; font-size: .6em;\">$v_file</div><br>"; } } close VIEWFILE; #=cut

Thanks a lot!

Replies are listed 'Best First'.
Re: Reading Text Lines
by davorg (Chancellor) on May 11, 2006 at 16:00 UTC

    The basic idea is to have the number of the first line that you want to display and use $. to work out where you are in the file.

    my $page = 1; # which page to display my $display_perl_page = 100; my $first = ($page - 1) * $display_perl_page; my $last = $first + $display_perl_page; while (<VIEWFILE>) { next unless $. >= $first; last if $. > $last; # display this record }

    (That code is untested, there may be an off-by-one error)

    There are modules like Data::Page that make this all a lot easier.

    Another alternative might be Tie::File.

    --
    <http://dave.org.uk>

    "The first rule of Perl club is you do not talk about Perl club."
    -- Chip Salzenberg

      And if you're going to do this kind of thing frequently it may make sense to build an index file with line numbers mapped to offsets from tell and then use seek to jump straight to that location in the file.

Re: Reading Text Lines
by CountOrlok (Friar) on May 11, 2006 at 16:06 UTC
    Your post isn't clear about how the user can choose which 100 lines to see. The following will show a specific 100 lines from the file:
    my $first_line_to_show = 427; open VIEWFILE, "< $path" or die "Can't open $path: $!<br>\n"; while( my $v_file = <VIEWFILE>) { next if $. < $first_line_to_show - 1; last if $. >= $first_line_to_show + 100; print "<div STYLE=\"font-family: 8pt Courier, 'Courier New', monos +pace; font-size: .6em;\">$v_file</div><br>"; } close VIEWFILE;
    -imran
Re: Reading Text Lines
by ruzam (Curate) on May 11, 2006 at 17:01 UTC
    seek() and tell() are what you need
    # get the last $pos from somewhere my $pos = 0; open VIEWFILE, "< $path" or die "Can't open $path: $!<br>\n"; seek (VIEWFILE, $pos, 0) or die "Couldn't seek to $pos: $!<br>\n"; my $count = 100; while( defined(my $v_file = <VIEWFILE>) and $count--) { print "<div STYLE=\"font-family: 8pt Courier, 'Courier New', mon +ospace; font-size:.6em;\">$v_file</div><br>"; } # save $pos somewhere for the next page view $pos = tell(VIEWFILE); close VIEWFILE;
    How you keep track of $pos and pass it from page to page is up to you.
      I thought of that, but the problem with tell+seek is that you can't go backwards unless you keep track of all the previous positions. A hybrid solution would do the trick.
      my $max_lines = 100; open(my $fh, '<', $path) or die("Unable to open file $path: $!\n"); my $line = $cgi->param('line') || 0; my $pos = $cgi->param('pos'); if (defined($pos)) { seek($fh, $pos, 0) or die("Unable to seek to $pos: $!\n"); } else { if ($line) { do { <$fh> } while $. < $line; } } my $lines = $max_lines; print(qq{<div class="lines">\n}); while ($lines-- && defined(my $line = <$fh>)) { chomp($line); print(html_escape($line), qq{<br>\n}); } print(qq{</div>\n}); my $first = ($line == 0); my $last = not defined(<$fh>); my $prev_line = $line - $max_lines; $prev_line = 0 if $prev_line < 0; my $prev = "?line=$prev_line"; my $next_line = $line + ($max_lines - $lines) - 1; my $next_pos = tell($fh); my $next = "?line=$next_line&pos=$next_pos"; if ($first && $last) { print(qq{No other pages.<br>\n}); } else { print(qq{<a href="$prev">[prev page]</a> }) if !$first; print(qq{<a href="$next">[next page]</a> }) if !$last; print(qq{<br>\n}); }
      would I be able to reach the end of the file like that?
        As long as you save $pos after the run, and set it up again at the beginning of the next run, you'll continue to read through the file starting where you left off.

        In my example, $pos is set to 0, but in reality you would need to set it to the value it had after the last run. Setting it to 0 each time would continually read the first 100 lines over and over. You could pass it as a hidden form variable so when the user clicks 'next' they get the next 100 lines of file. You could also pass the previous $pos as a hidden form variable so the user gets the previous 100 lines of file when they click 'back'.
Re: Reading Text Lines
by smokemachine (Hermit) on May 11, 2006 at 19:18 UTC
    could be this?
    perl -ne '$pages[$i].=$_;$i++unless$.%100' file

      Why keep 100MB in memory when you only need a few KB at a time?

      Why read the entire file when only displaying the first page?

      By the way,
      if$.%100==0
      is shorter and clearer than
      unless$.%100

        if$.%100==0 is shorter and clearer than unless$.%100

        I believe it's a matter of taste. My (maybe weird) way of thinking is that I'm interested in $a % $b when the modulo operator yields a false value, in the same way that I'm interested in $a == $b when the result is true. So, the alternatives are:

        if $a % $b == 0 if !($a % $b) if not $a % $b unless $a % $b

        To me, the clearest of them is the last one. Maybe it's because I use it regularly. Note that the one you said was shorter and clearer is the longest of that four, once we beautify the code.

        --
        David Serrano

        Sorry... 100mb? i didn't read this...
        :wq