clinton,
So hear is me talking out my @$$

Typically, what I would suggest is the prospector method. You pass the text through a series of finer filters until you find the nuggets you are looking for. Things that fall out or records that yield no gold are examined by hand to determine why. Either existing filters are modified or new ones are added. This can reduce the amount of manually work greatly.

The trouble with using this approach for this task is that even if it finds nuggets, you have to be sure you haven't been handed Fool's Gold. This is where using Bayesian and expert systems may help. Suppose you have more than just one series of filter and/or you have multiple dispositions (gold, pyrite, dirt, your wedding band, etc). By "ranking" the output, you can "teach" the system to recognize which filters and weights work best for which types of text.

Of course, you will never achieve 100% accuracy but I don't think you want to. Way over my head though.

Cheers - L~R

In reply to Re: Extracting structured data from unstructured text - just how difficult would this be? by Limbic~Region
in thread Extracting structured data from unstructured text - just how difficult would this be? by clinton

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.