Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses
 
PerlMonks  

comment on

( [id://3333]=superdoc: print w/replies, xml ) Need Help??
Be Spider Friendly!

Arggh! Building (or even running) a search engine is going to be a big effort and involve a fairly large amount of re-inventing of wheels. I work with one every day and keeping it up to date and working is a chore.

The quick and simple answer is that your best strategy will be to make your web site search engine friendly and then to piggy back of one of the commercial search engines (you don't have to use Google). A site is search engine friendly if the search engine spider has an easy time of getting around and doesn't have to follow too many deep links. Strangely enough, a site that is search spider friendly is also accessible by people who are visually disabled and using a text reader.

If you use an internet search engine, there is normally a search option that allows you to submit a search to an internet search engine such that it is limited to one domain (your website). If you have specific requirements, like wanting to ensure full coverage or having your own customised UI, you can pay a commercial search engine to index your site. My employers public website uses Atomz which seems to work well but is probably expensive.

The effort that you put into making your site spider friendly will pay dividends regardless of whether you implement a search engine yourself or get someone else to do it. The following links may be useful:

inman


In reply to Re: Building a search engine by inman
in thread Building a search engine by artist

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Are you posting in the right place? Check out Where do I post X? to know for sure.
  • Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
    <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
  • Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
  • Want more info? How to link or How to display code and escape characters are good places to start.
Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others goofing around in the Monastery: (None)
    As of 2024-04-25 01:45 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?

      No recent polls found