When I saw P2P i started thinking of a massive P2P effort to index the web...Use spare processor power from computers around the world to index the web. If you had that kind of power you could do more interesting indexs of documents, i wonder however if it could truly rival google or yahoo's indexers. Kind of like a dmoz.org or something. Just thinking out loud, don't mind me.

The problem that I see with a solution like this is that many would misuse it - depending on what part(s) of the system would be shared. I'm thinking of reverse engineering that could reveal how to be ranked better.

That's actually possible using existing horisontal scalability of HyperEstraier.

Just setup multiple servers which crawl separate parts of web. Setup search to search over all nodes at once.
Indexer can query search index to find out if some other indexer did crawl that page already (and optionally refresh content if needed). That way, you will have fresher pages with bigger number of incomming links (which you can count and use that also in page ranking - I hope that this idea doesn't violate Google patent).

I don't have pointer to perl solution for this (other than CPAN modules which make every problem 90% done). On the other hand, with current P2P architecture you can have multiple indexes (for e-mail, documents, etc.) and search over just some or all of them.


2share!2flame...

Why do you think that writing reverse index in Perl would be overkill? And in what meaning it would be an overkill?

Sure C/Perl combination might be considered - I could finally put C/C++ knowledge gained on advanced school to practical use ...

Only down-side to perl only version is speed. Of course, it depends on size of your input data. However, on my laptop I have more data that I want to index than any perl-only solution really can handle (over 20Gb in various formats).

I have some expiriences with WAIT (and some pending patches at http://svn.rot13.org/~dpavlin/svnweb/index.cgi/wait/log/trunk/ ), swish-e, Xapian (another great engine which updated perl bindings few days ago). I also experimented with CLucene perl bindings and finally ended with HyperEstraier.

I would suggent to make list of requirements of search engine and then select right one. My current list include:

• full text search
• filter results by attributes (e.g. date, category...)
• ability to update index content while running searches on it
• wildcard support (or substring, even better!)
• acceptable speed on projected amount of data
Last point influence choice very much. I would go with Plucene if data size is small enough (or only for prototyping).

Writing good parsers and analyzers for input formats (do you want to rank bold words more than surround text?) and font-end is hard enough without writing own reverse index implementation, especially since some very good allready exist.


2share!2flame...

In my experience, Plucene was not very good at handling your third requirement, "ability to update index content while running searches on it." The code that handles file locking is prone to die instead of wait. Not good for live websites. The following ASCII depicts my expression upon discovering this:

8-[

YMMV