in reply to Word Frequency in Particular Sentences

First, look for pdf in CPAN to find a module that parses PDF files.

Second, watch out for abbreviations, which usually end with a period, and sentences that end with something other than a period.

Third, it's Perl, not PERL; Perl is not an acronym.

emc

Information about American English usage here and here. Floating point issues? Please read this before posting.

  • Comment on Re: Word Frequency in Particular Sentences

Replies are listed 'Best First'.
Re^2: Word Frequency in Particular Sentences
by papidave (Pilgrim) on Mar 28, 2008 at 11:54 UTC
    swampyankee++ for noticing the problem with abbreviations. Short of a the ability to parse and comprehend grammar, it's going to be very difficult to separate

    "We sold the division to MegaTech, Ltd. in Asia last week, who flipped the sale to someone else."
    from
    "We sold the division to MegaTech Industries. In Asia last week, they flipped the sale to someone else."
    other than the fact that we are supposed to start a new sentence with an upper-case letter. There may be examples where that following word is a proper noun, however -- in which case it's going to be a very hard nut to crack.

    If, however, you only care about the "typical" case (because this is going to be a one-shot tool), you could:

    1. Split the text on /[.]\s+[A-Z]/ to get sentences.
    2. Grep the text for /[aA]sia/, or for /Asia\s/ if you don't want the word "asian" to count.
    3. Split the sentences that pass on ' ' to get words.
    4. Use the words you get from that split as keys to a hash, and increment a count in each bin.
    Q.E.D.

[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom