Extracting arbitrary "key phrases" is really a form of natural language parsing. To identify such key phrases you will need to develop a program that:

• splits text into words.
• identifies the part of speech associated with each word
• identifies phrases composed of the correct combinations of parts of speech. A non-exhasutive list of the correct combinations would look something like this:

    noun_phrase
       :: (adjective*) noun prepositional_phrase
    prepositional_phrase
       ::  preposition [article] noun_phrase
  [download]

Not a one of these are entirely trivial.

• Splitting into words.White space and punctuation will make a good first cut at splitting up text into words, but there will be important edge cases: hyphenated compound words in English are the first to come to mind. Sometimes hyphens are punctuation similar to a comma, sometimes they are minus signs, and sometimes they count as part of a compound word. If you care about capturing compound words in your keyword algorithm, your program will need to determine which one is which.
• Identifying parts of speech. Many words in English have multiple parts of speech depending on context. For example, "Johnny broke Nancy's nail file whilst trying to use it to nail nails.". In that single sentence the word "nail" is used as a noun, verb and adjective! Furthermore you will need to take into account all the forms of the noun. In English that is fairly simple - we have only four: singular, singular possessive, plural, plural possessive. In other languages the situation is much more complicated.
• Identifying runs of words that contain the correct part of speech combinations. I started this post with a partial description of the valid part of speech sequences. Once your text is tagged, this may well be the easiest part of the entire process.

As for modules that do that, I think you may be on your own. There are a few CPAN modules that can help you on your way, but they won't do the whole job for you. I've never used these but you may want to check out the various modules in the "Lingua" namespace:

• Part of speech tagging: Lingua::BrillTagger, Lingua::EN::Tagger
• Phrase extraction from tagged text: Lingua::YaTeA

Best, beth

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.