Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options
 
PerlMonks  

Expert Detection

by Tanalis (Curate)
on Oct 01, 2003 at 10:41 UTC ( [id://295564]=perlquestion: print w/replies, xml ) Need Help??

Tanalis has asked for the wisdom of the Perl Monks concerning the following question:

This post very much references Perl and Web Services as background.

The project that I am involved in/responsible for has been extended to allow the provision, to a group of medical students, of a newsgroup-style discussion forum, I guess not unlike PerlMonks, to give them the opportunity to explore and discuss topics they have been recently studying.

As part of this, the web service needs to provide some form of real-time chat (probably the only component of the project that'll end up using Java, at least client-side), allowing the students to ask questions of a panel of experts (university lecturers).

These experts can be selected in one of two ways: either with them being on the teaching staff (trivial to detect), or, in the case of an external expert, by their responses to questions.

This expert detection based on response requires two things: firstly, that the subject of the OP and its replies is known, something I feel would be fairly simple to detect, either by users selecting a category, or by some automated scripting. Secondly, and more challengingly, the responses need to be analysed in some way to allow the system to detect that the replying expert is answering the questions posed to him. Quality control is a fairly large (and currently unresolved) issue here.

I've been investigating a number of ways to try and provide a form of expert detection, the most simple of which appears to be a set of regular expressions, all with weightings, to differentiate between questions and answers (SpamAssassin style). This has flaws, however, and as mentioned before, the quality of replies is not taken into account at all.

Does anyone have any experience with this form of text analysis? Any suggestions that can be offered into methods for carrying this out would be appreciated - as would any thoughts on the quality control issue.

Thanks in advance for your help.

Replies are listed 'Best First'.
Re: Expert Detection
by tachyon (Chancellor) on Oct 01, 2003 at 11:22 UTC

    As Abigail-II points out how is a computer supposed to do this? This is AI++. Take A-II's posts for example. Few here would doubt that A-II is indeed an expert, but the content delivered in a given A-II reply depends on all sorts of things from the quaility of the question through to the phases of the moon. Regardless of this fact very few users here have any problem recognising that A-II is indeed an expert.

    Why?

    Humans are really good at heuristic analysis, so use the force. Don't try to do it the hard way. Let your expert heuristic humans identify the subject experts for you with some sort of rating/XP type system. They will do it far more accurately than anything you are likely to knock up in the forseeable future.....You are trying to compete with computers far more complex than what we currently have readily available, each with 20+ years of training in dealing with the nuances of language.

    cheers

    tachyon

    s&&rsenoyhcatreve&&&s&n.+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print

Re: Expert Detection
by Molt (Chaplain) on Oct 01, 2003 at 11:00 UTC

    If I'm not misunderstanding you I think what you're asking is bordering the realm of Hard AI, where the computer needs to understand the topic that you're discussing.. and were you to solve that slight problem you'd end up not needing the external experts at all as the computer could answer. If you do manage this though let me know and I'll, uhh, market it for you.

    Rather than trying to solve this with text analysis I'd recommend you select a selection of internal people who you can trust and who can 'promote' someone up to Expert level based on their perception of their responses. Admittedly a clever troll may be able to baffle them with science, as Magnus Pike used to say, but that's pretty much a problem inherant in the system to be honest.

    If you're willing to expose your students to one of the more dangerously time-wasting potential technologies you may want to look into some kind of passworded IRC system for the real time chat. IRC is a stable and mature system with the added benefit that it's got a fairly flexible security system, and that with a few well-placed bots the students have access to data-retrieval and conversation archiving systems. A bot could also be used to restore the priviledged status of the above students should you take that route.

Re: Expert Detection
by Abigail-II (Bishop) on Oct 01, 2003 at 11:01 UTC
    So, your question is "out of a large group of people chatting, some possibly lying, detect the expert"? This is a hard problem - kind of the reverse the Turing test. I could imagine this being a ph. D. research problem. None the less, I'd also expect a great deal of research has already be done (and published) about this problem. So you might want to take a trip to the university library.

    Personally, I'd solve the problem in a totally different way. When I arrange the details for the expert, like when and where to log in, I'd give him a user-id/handle/nick/whatever. Then I know who the expert is before hand.

    Abigail

Re: Expert Detection
by tilly (Archbishop) on Oct 05, 2003 at 03:20 UTC
    My suggestion is that the best likely approach is to recognize people based on how other people react to them. This is like how Google decides what pages are good - if lots of people recommend it, then it is likely to be good.

    There is lots of research on community dynamics and how to sort them out based on logs of the research. Unfortunately while I know that it is out there, and I know some places that use it, I don't know much about it (eg good starting places to recommend), and I do not know how much of it is public vs locked up as trade secrets of places like Google.

    Therefore the best advice that I can give is to write a letter like this:

    I would like to write a system that identifies people who are likely to be experts in an online chat environment based on the dynamics of the interactions between people. I know that there has been at least some work on community dynamics which could be relevant to this, and I have heard enough about you to suspect that you might be able to give me pointers on where I should start researching the feasibility of this.
    And then send it to people whose publically documented interests suggest some level of interest in topics like this. Off of the top of my head, I would think that Clay Shirky is an obvious candidate either for knowing this, or for dropping names of people who might know this.

    Good luck Googling for names to ask, and good luck in approaching them.

Re: Expert Detection
by saintbrie (Scribe) on Oct 02, 2003 at 01:42 UTC

    This is a really fascinating question. Contrary to what the other posters have said so far, a rudimentary system for expert detection is not so far off. Your idea about using textual analysis to arrive at an answer is a good one. You might actually have a go at it using the AI::Categorizer modules.

    The drawback to this approach is that you are going to want to have a large body (several hundred) of answers for each question, each judged by several experts to be either expert or not expert. Just as you suggest spam assassin does with spam, you could do with answers.

    This kind of automation is something that is going on right now grading college composition students essays.

    Yes, someone might be able to wheedle past a couple of the questions, but my guess is that it is going to eliminate the vast majority of those who try.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://295564]
Approved by broquaint
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others surveying the Monastery: (6)
As of 2024-03-28 10:41 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found