Beefy Boxes and Bandwidth Generously Provided by pair Networks
Think about Loose Coupling
 
PerlMonks  

comment on

( #3333=superdoc: print w/replies, xml ) Need Help??

The fact that WordNet only contains open-class words might be a benefit, actually. If it's not in WordNet, the OP probably doesn't want it in the concordance.

...except that that idealizes WN's coverage of English. You'd probably do well, though with the added heuristic: "and it's a short, uncapitalized word". (i.e. "Short, uncapitalized words that aren't in WN should probably be ignored.")

...and it ignores the problem of homonyms. (e.g. 'in' has 7 senses in WordNet)

Nonetheless, felt the need to "defend" WN (since someone's quoting from the FAQ I wrote).

Better solutions have been mentioned elsewhere in the thread. But, I'll also add that my approach to the problem would probably be more Information Retrieval oriented. I'd use KStem, a stemming algorithm whose output is (in the usual case) an actual word. It tries to correct some of the problems with Porter. That paper is pretty academic (but good). There's also downloadable Java code.

Then I'd just use tf-idf style weights to pick out the interesting words.

Seems easier than parsing, even if it's shallow parsing.


In reply to Re: Trying to set up a concordance using linguistic rules by benizi
in thread Trying to set up a concordance using linguistic rules by Quicksilver

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Are you posting in the right place? Check out Where do I post X? to know for sure.
  • Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
    <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
  • Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
  • Want more info? How to link or or How to display code and escape characters are good places to start.
Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others lurking in the Monastery: (3)
As of 2022-09-27 20:14 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    I prefer my indexes to start at:




    Results (124 votes). Check out past polls.

    Notices?