Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical

Re^5: Running SuperSearch off a fast full-text index.

by creamygoodness (Curate)
on Jun 11, 2007 at 18:58 UTC ( #620555=note: print w/replies, xml ) Need Help??

in reply to Re^4: Running SuperSearch off a fast full-text index.
in thread Running SuperSearch off a fast full-text index.

Does this mean that for mod_perl running the prefork MPM, each child process needs to load the cache? That must use a lot of memory, no?

Yes, and KinoSearch is not thread safe. The memory requirements can be significant for large indexes, even though the data structures are not Perl's and attempts have been made to keep things compact.

And how do you handle cache updates across all the child processes (whether they're on the same machine or different machines?

A Searcher instance represents a snapshot of the index in time. Until you manually reload by creating a new Searcher, changes to the index are not visible.

Marvin Humphrey
Rectangular Research ―
  • Comment on Re^5: Running SuperSearch off a fast full-text index.

Replies are listed 'Best First'.
Re^6: Running SuperSearch off a fast full-text index.
by clinton (Priest) on Jun 11, 2007 at 19:09 UTC
    So maybe a reasonable solution would be:
    • a separate mod_perl search server, which takes search requests from the web server and returns (eg) an XML or Soap list of IDs
    • each child process checks (eg) a last_cache_update file once a minute to decide whether to reload the caches or not


      Yes, that'll work. Tip: if you're fetching docs from a DB rather than from the index, you may want to turn off stored and vectorized for your fields to save disk space, though that will prevent you from using KinoSearch's built-in Highlighter. Those properties are on by default because disk space is cheap and stuff should be easy.
      Marvin Humphrey
      Rectangular Research ―

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://620555]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others avoiding work at the Monastery: (4)
As of 2021-10-21 10:36 GMT
Find Nodes?
    Voting Booth?
    My first memorable Perl project was:

    Results (83 votes). Check out past polls.