in reply to How to reverse a huge file in Perl?

You can use Tac (Unix) if you are on a Unix like operating system. It's much faster than using Perl and File::ReadBackwards:
# ls -lh /var/lib/mysql/ibdata1 -rw-rw---- 1 mysql mysql 66M Mar 19 11:05 /var/lib/mysql/ibdata1 # wc -l /var/lib/mysql/ibdata1 323833 /var/lib/mysql/ibdata1 # time tac /var/lib/mysql/ibdata1 > /tmp/bar real 0m0.643s user 0m0.173s sys 0m0.322s # time /usr/bin/perl > /tmp/foo use warnings; use strict; use File::ReadBackwards; my $bw = File::ReadBackwards->new( '/var/lib/mysql/ibdata1' ) or die "$!" ; while( defined( my $log_line = $bw->readline ) ) { print $log_line ; } __END__ real 0m29.076s user 0m11.910s sys 0m10.819s # md5sum /tmp/foo 2b31d9f47525853842d5dbce584bd95c /tmp/foo # md5sum /tmp/bar 2b31d9f47525853842d5dbce584bd95c /tmp/bar
Update Wed Mar 26 10:05:03 CET 2008: Added number of lines in input file.

Update Wed Mar 26 10:11:31 CET 2008: It seems that the tac + process file forwards using Perl approach combined is still much faster than process file using Perl and File::ReadBackwards:

# time /usr/bin/perl > /tmp/foo use warnings; use strict; use File::ReadBackwards; my $bw = File::ReadBackwards->new( '/var/lib/mysql/ibdata1' ) or die "$!" ; while( defined( my $log_line = $bw->readline ) ) { $log_line =~ s/1/2/g; print $log_line; } __END__ real 0m27.431s user 0m12.906s sys 0m10.701s [root@afflinux aff]# time /usr/bin/perl > /tmp/bar use warnings; use strict; my $FH = undef; open($FH, '/var/lib/mysql/ibdata1' ) or die "$!" ; while( my $log_line = <$FH> ) { $log_line =~ s/1/2/g; print $log_line; } __END__ real 0m5.249s user 0m1.535s sys 0m0.293s
--
Andreas

Replies are listed 'Best First'.
Re^2: How to reverse a huge file in Perl?
by BrowserUk (Patriarch) on Mar 26, 2008 at 09:02 UTC
    It's much faster than using Perl and File::ReadBackwards:

    True if the only aim is to reverse the file.

    If however, the aim is to process the data in the file in the reverse order using perl, you'd have to compare F::RB with using tac and then reading every record in the file using Perl.

    Also, how many lines are there in your test file?


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
Re^2: How to reverse a huge file in Perl?
by samtregar (Abbot) on Mar 26, 2008 at 17:02 UTC
    Wow. Might be interesting to take a peak at the tac source code to try to figure out what it's doing that's so much better than File::ReadBackwards. There must be a trick here, that's way more than the standard C-versus-Perl difference.

    -sam

        Didn't he already show you a wc -l? Did you test it yourself and get different results?

        -sam

        If the file he used doesn't contain any newlines

        Obviously, the file does contain 323833 newlines.

Re^2: How to reverse a huge file in Perl?
by Arunbear (Prior) on Mar 27, 2008 at 00:00 UTC
    tac is also available for win32 e.g.
    C:\mycode>perl -le "print $_ for qw[spam bacon eggs]" > things.txt C:\mycode>cat things.txt spam bacon eggs C:\mycode>tac things.txt eggs bacon spam