Re: Extracting structured data from unstructured text - just how difficult would this be?

clinton,
So hear is me talking out my @$$

Typically, what I would suggest is the prospector method. You pass the text through a series of finer filters until you find the nuggets you are looking for. Things that fall out or records that yield no gold are examined by hand to determine why. Either existing filters are modified or new ones are added. This can reduce the amount of manually work greatly.

The trouble with using this approach for this task is that even if it finds nuggets, you have to be sure you haven't been handed Fool's Gold. This is where using Bayesian and expert systems may help. Suppose you have more than just one series of filter and/or you have multiple dispositions (gold, pyrite, dirt, your wedding band, etc). By "ranking" the output, you can "teach" the system to recognize which filters and weights work best for which types of text.

Of course, you will never achieve 100% accuracy but I don't think you want to. Way over my head though.

Cheers - L~R

Comment on Re: Extracting structured data from unstructured text - just how difficult would this be?

Replies are listed 'Best First'.

Re^2: Extracting structured data from unstructured text - just how difficult would this be?
by clinton (Priest) on Feb 21, 2008 at 16:51 UTC

I was thinking about something along exactly these lines, so we may just be two talking @$$'$

What'd be interesting is trying to look for "contextual words", so does May refer to the month or the daughter, London is a place, or Jack London. It would be impossible to predict all of these ambiguities, so the "training" makes a lot of sense to me.

Of course, you will never achieve 100% accuracy but I don't think you want to.

Absolutely correct - we don't depend on this data, it just adds value when we can extract it.

thanks for the input

Clint

[reply]