s/[^a-z]/ /g;
@words= split ' ';
[download]

1. words that are broken at a line-
   end ;-)
2. foreign language characters
3. your definition of a word. Maybe you render ab4711xya word. With this it will be 2.

This also breaks for simple, commonly used words. Like "don't." That's two words for you right there. Or "co-worker." Or...

I'd suggest that the simplistic approach would be better off by splitting on white space first, then simply removing non-word characters (and I'd use the [:alpha:] designation as opposed to a-z).

Basically, you have to take whitespace-separated words, removing non-alphanumeric (i.e. punctation) characters from the beginning or the end. Punctation is allowed inside a word. What makes this a bit more complicated is that in some languages, the space os omitted around em dashes, so you have to consider em-dashes as whitespace too. Thus, it's easier to define what separates words than what counts as a word. Here's a simple solution.

#!perl

use warnings;
use strict;

my # I hate to say this, as it's not mine
$text = q{

"'My Dear Mr. Sherlock Holmes:--"Lord Backwater tells me that I may
place implicit reliance upon your judgement and discretion. I have
determined, therefore, to call upon you and to consult you in referenc
+e
to the very painful event which has occurred in connection with my
wedding. Mr. Lestrade, of Scotland Yard, is acting already in the matt
+er,
but he assures me that he sees no objection to your co-operation,
and that he even thinks that it might be of some assistance. I will
call at four o'clock in the afternoon, and, should you have any other
engagement at that time, I hope that you will postpone it, as this
matter is of paramount importance. Yours faithfully, ST. SIMON.'

 -- Sir Arthur Conan Doyle, The Adventures of Sherlock Holmes. 1892.

};

use locale;
my $stuff_between_words = qr{
  [^[:alnum:][:space:]]*
  (?: [[:space:]]+ | --+ | ^ )
  [^[:alnum:]]*
 }x;
my @words = split $stuff_between_words, $text;
shift @words; # remove the empty word from the beginning

print join("  ", map "<<$_>>", @words), "\n";

__END__
[download]

However, if I'd really need a fast solution, I'd use flex.

Update: on second thought, it'd be better to use just -- instead of --+.

Thanks I will take a look at flex.

use strict;
use warnings;

my $stripped = qr/["!]/;
my $word = '"Wilmer!"';
$word =~ s/^(?:$stripped)+(.*?)(?:$stripped)+$/$1/;
print $word;
[download]

After looking at your regexp I took to simplifying my needs with:

$word =~ s/^[^\w\d]+(.*?)[^\w\d]+$/$1/;

My intention is remove everything that is not a letter or number up to the first letter, pull everything up till the last non letter or digit. When I look at it it makes sense, but my testing it does not work.

Update

It works on the simple example I gave for 'Wilmer!'. I was running word count with a script as the input and the odd results I was seeing were the syntax in the script. I apologize.

Except you want to strip punctuation from the beginning or end. The above regex only works if there is punctuation at both beginning and end.

If removing any trailing/leading punctuation is in fact your goal, what about something like:

use strict;
use warnings;

my $word = 'Wilmer",';

$word =~ s/^  \W*?          # ignore any leading punc 
            ( \w .*? )      # swallow everything lazily
            (?: \W+  )? $   # ignore any trailing punc
          /$1/x;            

print $word;
[download]

Update: Mind you, at that point, a much simpler regex will likely serve you better in terms of speed and readability:

$word =~ s/(?:^\W+)|(?:\W+$)//g;
[download]

Final update - benchmark:

               Rate     capture non_capture
capture     16561/s          --        -28%
non_capture 22861/s         38%          --
[download]

The second suggestion is about 30% faster, on average.

Additionally, \w doesn't mean what you think it means.

#! perl
use strict;
$_="Wilmer'?";

tr/a-zA-Z//csd;
print;
[download]

GNU's wc command exists for any major platform. Simply call it from inside your Perl application and let it do the heavy lifting.

This does a total word count to my understanding. I am looking for individual word counts. I did not mention that though, but thanks anyways.

the user define what punctuation they want removed

If it's important for your application to allow the user to specify a "cusomized" set for some particular case, you face a variety of tricky issues:

• Is it easier for the user to specify the particular non-alphanumerics that should be kept, rather than specify all the ones that should be removed? (I think maybe so.)
• Many of the characters involved have special meanings in regexes (period, question-mark, plus, dollar-sign and some others). This isn't a killer, but you need to be mindful of it.
• If some "non-letter" characters are to be kept when they occur at a word boundary, you might have problems when other "non-letters" (that are to be discarded) co-occur with the ones being kept.

  For example, suppose that hyphen is to be kept at word boundaries, but parens at word boundaries should be removed; in a string like " word)- " it will be hard to remove the paren, because it lies "inside" the hyphen, which is being retained; that paren is "word-internal". Maybe the user just needs to specify which non-letter characters to keep when they occur next to a letter (or maybe it's more complicated than that).

Anyway, in the default case, it really can be very simple (and this might even be the quickest):

use strict;

# make up some data
my $line = "('The text.')-- 5 o'clock! What's cookin' with the text da
+ta?";

# split on whitespace, keep only tokens that contain at least on lette
+r
my @words = grep /\p{L}/i, split ' ', $line;

my %wcount;
for ( @words ) {  # using $_ will modify @words "in-place"
    s/^\P{L}+//;  # remove initial non-letters
    s/\P{L}+$//;  # remove final non-letters
    $wcount{lc()}++;  # normalize to lower-case-only
}

print "$wcount{$_}\t$_\n" for ( sort keys %wcount );

__OUTPUT__
1       cookin
1       data
1       o'clock
2       text
2       the
1       what's
1       with
[download]

Some things which often fail to get taken into account:

Words with internal apostrophes. (don't won't, can't shouldn't, you'll, it's, etc.)

Words with non ASCII characters. á, ñ, ÿ, etc.

This does:

##########################################################
#! /usr/bin/perl
use warnings;
use strict;

my $word = qr/(?<!\p{Alnum})\p{Alnum}+(?!\p{Alnum})/;
my %count;

while (<DATA>) {
    my $line = lc $_;
    while ($line =~ /($word('$word)?)/g){
        $count{$1}++;
    }
}

printf "%15s %5d\n", $_, $count{$_} for sort keys %count;

__DATA__
"Hello World!"
"Oh poor Yorick, his world I knew well yes I did"
"Words with internal apostrophes. (don't won't, can't shouldn't, you'l
+l, it's, etc.)"
"Señor Montóya's resüme isn't ápropos."
[download]

The ultimate case would be to handle a word count in multiple languages, btu I believe that there are too many factors to take into account. Thanks for the suggestion though.

$word =~ s/_//g;
$word =~ /(\w+)/ and $word = $1;
[download]