Perhaps the answer isn't so complicated. It would be crazy to do a full-content comparison across your whole database, but it will probably be sufficient to single out a few special cases and likely aliases and just test for those. in the case you mention - www.foo.com eq foo.com - it would be very easy to look out for just that pair and checksum the two pages to make sure they differ.
incidentally, since you're doing something clever and complicated to the content of each item later on anyway, perhaps you'd be better off just grabbing everything and deferring redundancy checks until you start munching it up?
in other words, my .02p says this situation requires laziness, not hubris.
-
Are you posting in the right place? Check out Where do I post X? to know for sure.
-
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big>
<blockquote> <br /> <dd>
<dl> <dt> <em> <font>
<h1> <h2> <h3> <h4>
<h5> <h6> <hr /> <i>
<li> <nbsp> <ol> <p>
<small> <strike> <strong>
<sub> <sup> <table>
<td> <th> <tr> <tt>
<u> <ul>
-
Snippets of code should be wrapped in
<code> tags not
<pre> tags. In fact, <pre>
tags should generally be avoided. If they must
be used, extreme care should be
taken to ensure that their contents do not
have long lines (<70 chars), in order to prevent
horizontal scrolling (and possible janitor
intervention).
-
Want more info? How to link
or How to display code and escape characters
are good places to start.
|
