From a file containing a word every line, I need to create n-grams of 2 (nad 3 and 4).

Input is

Hello
I
am
happy

[download]

Output should be for n-grams of 2

Hello I
I am
am happy

[download]

I'm using the following code (this is for n-grams of 2), which works fine exept a) it doesn't look nice b) is quite slow for big input files (expecially for n-grams of 4)

my $input= "input.txt";
my $output = "output.txt";
open (OUT, ">$output") || (die "Can't open $output\n");
open (IN,"$input") || (die "WARNING: $input not found\n");

    my $line = <IN>;
    while($line){
        chomp $line;
        push(@token, $line);
        $line = <IN>;
    }
    
    my $shift=2;
    my $counter=0;
    my $TokenTotal = @token;
    while($counter <= ($TokenTotal-$shift)){
        print OUT "$token[$counter]\t$token[$counter+1]\n";
        $counter++;
        $line = <IN>;
    }
close IN;
close OUT;
}

[download]

I am aware there is a Text::ngrams available, but I'd like to have a solution which I can perfectly control. Any suggestion and improvement is appreciated.

In reply to Create n-grams from tokenized text file by Anonymous Monk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.