You can't do it precisely programmatically. You have to determine that two pages are "close enough" when you hit them. I know Google knows to do that, but I've run into other web walkers that don't.
I found that out by putting a link on my webserver root page to -. And I symlinked "-" to "." in my root doc directory. So any page on my website was accessible by any number of /-/-/-/-/- prefix chars before the real URL. Google immediately figured it out, but I had other webcrawlers visiting (and indexing!) my entire web site some 15 or 20 times deep before giving up.
If you are spidering your own site, you can add code in your spider to canonicalize your URLs before fetching. I did that in a few of my columns.
-- Randal L. Schwartz, Perl hacker
Be sure to read my standard disclaimer if this is a reply.