Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?
 
PerlMonks  

Re^2: Trying to set up a concordance using linguistic rules

by Not_a_Number (Prior)
on Feb 25, 2008 at 18:37 UTC ( #670082=note: print w/replies, xml ) Need Help??


in reply to Re: Trying to set up a concordance using linguistic rules
in thread Trying to set up a concordance using linguistic rules

Given that WordNet explicitly excludes "determiners, prepositions, pronouns, conjunctions, and particles", I don't really see how a module providing an interface to this (otherwise excellent) resource might help the OP.

  • Comment on Re^2: Trying to set up a concordance using linguistic rules

Replies are listed 'Best First'.
Re: Trying to set up a concordance using linguistic rules
by benizi (Hermit) on Feb 28, 2008 at 05:13 UTC

    The fact that WordNet only contains open-class words might be a benefit, actually. If it's not in WordNet, the OP probably doesn't want it in the concordance.

    ...except that that idealizes WN's coverage of English. You'd probably do well, though with the added heuristic: "and it's a short, uncapitalized word". (i.e. "Short, uncapitalized words that aren't in WN should probably be ignored.")

    ...and it ignores the problem of homonyms. (e.g. 'in' has 7 senses in WordNet)

    Nonetheless, felt the need to "defend" WN (since someone's quoting from the FAQ I wrote).

    Better solutions have been mentioned elsewhere in the thread. But, I'll also add that my approach to the problem would probably be more Information Retrieval oriented. I'd use KStem, a stemming algorithm whose output is (in the usual case) an actual word. It tries to correct some of the problems with Porter. That paper is pretty academic (but good). There's also downloadable Java code.

    Then I'd just use tf-idf style weights to pick out the interesting words.

    Seems easier than parsing, even if it's shallow parsing.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://670082]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others examining the Monastery: (4)
As of 2022-10-06 14:33 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    My preferred way to holiday/vacation is:











    Results (26 votes). Check out past polls.

    Notices?