If your data is easy to split on TAB's, perl's split is exceptionally fast. Building on BrowserUk's example - which is a good example - look at this test:

$ perl -E'say join"\t",qw(1 aap 2 noot 3 mies 4 wim 5 zus 6) for 1..2_
+000_000' > test.tsv
$ wc -l test.tsv
2000000 test.tsv
$ cat test.pl
use 5.016;
use warnings;

use Benchmark qw( cmpthese );
use Text::CSV_XS;

my $csv = Text::CSV_XS->new ({ sep_char => "\t", auto_diag => 1 });

cmpthese (3, {
    none        => sub {
        open my $fh, "<", "test.tsv";
        while (<$fh>) {
            }
        },
    splt        => sub {
        open my $fh, "<", "test.tsv";
        while (<$fh>) {
            my @f = split m/\t/ => $_, -1;
            }
        },
    tcsv        => sub {
        open my $fh, "<", "test.tsv";
        while (my $f = $csv->getline ($fh)) {
            # use $f->[1] instead of $f[1];
            }
        },
    tcsvbc      => sub {
        open my $fh, "<", "test.tsv";
        my @f = ("") x 11;
        $csv->bind_columns (\(@f));
        while ($csv->getline ($fh)) {
            }
        },
    });
$ perl test.pl
       s/iter   tcsv tcsvbc   splt   none
tcsv     5.26     --   -32%   -36%   -96%
tcsvbc   3.57    48%     --    -5%   -93%
splt     3.38    56%     6%     --   -93%
none    0.233  2156%  1429%  1347%     --
$

[download]

where vsespb is missing the point, is that when determining if a record is to be split or not, one still has to look at the record. That is exaclty what BrowserUk was pointing at. Looking at the first character of a record is a cheap operation, not needing split at all, but if you want to decide processing on the content of the 11th field, you have to split before you can decide.

All of the above will start changing when the fields can be quoted and have (valid) TAB characters inside the field, or if fields are allowed to have embedded newlines or when the fields are (much) longer. Compare to e.g.:

$ perl -E'@x=("x" x 100) x 11;say join"\t",@x for 1..2_000_000' > test
+.tsv
$ perl test.pl
       s/iter   tcsv tcsvbc   splt   none
tcsv     23.3     --   -32%   -74%   -92%
tcsvbc   15.8    47%     --   -61%   -89%
splt     6.13   280%   158%     --   -71%
none     1.78  1206%   788%   244%     --

[download]

Enjoy, Have FUN! H.Merijn

In reply to Re: Optimise file line by line parsing, substitute SPLIT by Tux
in thread Optimise file line by line parsing, substitute SPLIT by go9kata

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.