How about something like:

my ($word1, $word2) = split(/\s+/);
if defined($word2) {
   ...
}
[download]

instead of:

foreach $word (split /\s+/) {
   ...
}
[download]

In other words, don't loop over every word. Get the first two words of every line, and work with those.

Interesting ... I had to waste some time on this one. ;)

Now then, if you do perform a $word =~ s/\W//g; on the entire sentance, how do you know when the sentance ends? You have to keep some punctuation around. Anyways, try this out. Hopefully some other monks will have better answers for you, but this is a simple approach.

#!/usr/bin/perl -l
 
use strict;
use warnings;
use Data::Dumper;
 
my $data = do {local $/;<DATA>};
my @sent = split /[.!?]\B/,$data;
 
my @parsed;
for my $i (0 .. $#sent) {
    next if $sent[$i] =~ /^$/;
    my @word = map $_ || (), split /\s+/,$sent[$i];
 
    for (my $j = 0; $j < @word; $j += 2) {
        push @{ $parsed[$i] }, [ $word[$j], $word[$j+1] ];
    }
}
     
print Dumper \@parsed;
 
# second pair from second sentance (should be 'a test')
print join ' ', @{$parsed[1]->[1]};
 
__DATA__
This is Joe. This is a test. This is not a test.
Blah blah isn't this fun? I wish you were here! Nah ...
[download]

L-LL-L--L-LL-L--L-LL-L--
-R--R-RR-R--R-RR-R--R-RR
B--B--B--B--B--B--B--B--
H---H---H---H---H---H---
(the triplet paradiddle with high-hat)

use strict;

my (@lines, @words, $i, $pair, %hash);
foreach (<DATA>) {
    $_ =~ s/[^\w\. ]//g;          # Remove unneeded characters
    $_ =~ s/  +/ /g;              # Many spaces to one
    $_ =~ s/ ?\. ?(\. ?)*/\./g;   # Boundaries with . change to .
    $_ =~ s/^ //; $_ =~ s/ $//;   # Spaces at start and end removed
    $_ = lc($_);                  # Lowercase
    @lines = split(/\./, $_);     # Split on sentence boundaries
    foreach (@lines) {            # Get words for each sentence
        @words = split(/[\. ]/, $_);
        for ($i = 0; $i < $#words; $i++) {   # For each word pair
            $pair = @words[$i] . ' ' . @words[$i+1];
            $hash{$pair}++;       # Increment count for word pair
        }
    }
}
foreach (sort {$hash{$b} <=> $hash{$a}} keys %hash) {
    print $_ . ' ' . $hash{$_} . "\n";
}

__DATA__
Four score and seven years ago our fathers brought forth, upon this co
+ntinent, a new nation, conceived in liberty, and dedicated to the pro
+position that "all men are created equal"

Now we are engaged in a great civil war, testing whether that nation, 
+or any nation so conceived, and so dedicated, can long endure. We are
+ met on a great battle field of that war. We have come to dedicate a 
+portion of it, as a final resting place for those who died here, that
+ the nation might live. This we may, in all propriety do. But, in a l
+arger sense, we can not dedicate -- we can not consecrate -- we can n
+ot hallow, this ground -- The brave men, living and dead, who struggl
+ed here, have hallowed it, far above our poor power to add or detract
+. The world will little note, nor long remember what we say here; whi
+le it can never forget what they did here.

It is rather for us, the living, we here be dedicated to the great tas
+k remaining before us -- that, from these honored dead we take increa
+sed devotion to that cause for which they here, gave the last full me
+asure of devotion -- that we here highly resolve these dead shall not
+ have died in vain; that the nation, shall have a new birth of freedo
+m, and that government of the people by the people for the people, sh
+all not perish from the earth.
[download]

Hmm. So what you want is the first word pair in each sentence - but a count for that pair across all sentences?

I'm sort of confused by this statement. Is there any criteria for selecting a word pair? Or do you always want to select the first two words on a line as the word pair?

bob,
You have a hard problem. It is easy for a human to see that those two headlines are related, but a program only does what you tell it. One approach may be:

For each headline -
• Create a 2 element array of first two words and entire headline
• Go through all previous full headlines to see if it has been seen already
• If yes - increment the count, if no - add it as a new item
The problem is that there is likely a high probability that two words will be present in two different headlines that are not related. Other approaches might be to split out the words, sort them, and look for the total number in common. In any case, you are not going to come up with a fool proof system. If the logic above is what you want and you can't figure it out, let me know and I can whip up something.

Cheers - L~R