Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?
 
PerlMonks  

Re: Eliminating "duplicate" domains from a hash/array

by thpfft (Chaplain)
on Mar 31, 2003 at 14:48 UTC ( [id://246933]=note: print w/replies, xml ) Need Help??


in reply to Eliminating "duplicate" domains from a hash/array

Perhaps the answer isn't so complicated. It would be crazy to do a full-content comparison across your whole database, but it will probably be sufficient to single out a few special cases and likely aliases and just test for those. in the case you mention - www.foo.com eq foo.com - it would be very easy to look out for just that pair and checksum the two pages to make sure they differ.

incidentally, since you're doing something clever and complicated to the content of each item later on anyway, perhaps you'd be better off just grabbing everything and deferring redundancy checks until you start munching it up?

in other words, my .02p says this situation requires laziness, not hubris.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://246933]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others rifling through the Monastery: (6)
As of 2024-03-28 19:12 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found