I'm looking at similar ways to categorize and mine unstructured text. One commercial product we tried (none mentioned in the article) worked for smaller sets of data (under one million) and under 2000 categories. Beyond that is didn't scale well at all, as memory utilization grew considerably and CPU remained as the primary bottleneck. One job with 10,000 categories we started on a fairly new Sun box used 30GB RAM and was projected to take 570 days to finish!

Using a linux cluster, this looks to be much more reasonable for significantly larger data sets. By giving sets of source articles to each cluster node and storing results in various database tables, the jobs of parsing, tagging, tokenizing, and categorizing may be divided into relatively independent jobs. The only dependency so far might be in database connections, but hopefully connection pooling can take care of that. One of these days I mean to write up the whole thing as a type of perl data mining RFC. (Sound useful?)

Some of the ideas mentioned in the article are new to me. From the article:

"PowerDrill takes the unstructured data, namely sentences, and diagrams the sentences placing each part of speech, such as noun phrase, verb phrase, and prepositional phrase, into a separate field, actor, action, and object which can then be used by a standard database to discover relationships and trends."

So instead of, or maybe in addition to, using a term list, dictionary, and thesaurus, they are trying to extract knowledge based on sentence structure. That might be hard on some forums. Most teen forums I've seen involve many abbreviations, spelling errors, and slang. Some topic area forums aren't much better. That's a tough problem. In addition to that, written speech from books like Moby Dick is quite different from the written style used in most forums, just like written speech is very different from spoken speech.

Adding this kind of knowledge extraction to standard categorization and parsing would be a neat thing. It does lead to other questions:

• Where this is useful and what questions does it answer?
• On what types of source data does this work well?
• How scalable is their application as well as the technology in general?
• What kind of hardware does it require?

I would welcome stories from other monks about projects doing data mining, text processing, using clusters, or any related topics. I'm definitely still learning in this area, but with the contributions and help of people here we should be able to get a good handle on it all and explore interesting areas.

---

---

• Are you posting in the right place? Check out Where do I post X? to know for sure.
• Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:

  <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>

• Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
• Want more info? How to link or How to display code and escape characters are good places to start.