in reply to Create n-grams from tokenized text file
Hi Anonymous,
You're reading your entire file into memory, which isn't necessary because it looks like all you want is a sliding window over a certain number of lines from the file. This can be done by pushing each element onto the end of an array and then shifting elements off the beginning. This can even be done in a one-liner - see perl's -n and -l switches in perlrun:
$ perl -wnle 'push @x, $_; if(@x>=2) {print "@x"; shift @x}' input.txt Hello I I am am happy $ perl -wnle 'push @x, $_; if(@x>=3) {print "@x"; shift @x}' input.txt Hello I am I am happy
Another solution might be the core module Tie::File. Note I haven't tested this for efficiency, but since the module doesn't read the entire file into memory this solution should still be better than doing that. The module does have to scan the file once to get the number of records contained within:
#!/usr/bin/env perl use warnings; use strict; use Tie::File; use Fcntl 'O_RDONLY'; die "Usage: $0 INPUTFILE WINSIZE\n" unless @ARGV==2; my ($INPUTFILE,$WINSIZE) = @ARGV; tie my @array, 'Tie::File', $INPUTFILE, mode => O_RDONLY; $, = " "; $\ = "\n"; # output field/record separators for my $i (0..@array-$WINSIZE) { print @array[$i..$i+$WINSIZE-1]; }
Example usage:
$ perl window.pl input.txt 2 Hello I I am am happy
Update: My first solution is basically a one-liner version of Marshall and LanX's solutions, and my second solution is quite similar to BrowserUk's solution. Also switched from using print "@...\n" to setting $, and $\.
Hope this helps,
-- Hauke D
|
|---|