in reply to Re: Test::WWW::Mechanize page_links_ok fails on wikipedia entry external links
in thread Test::WWW::Mechanize page_links_ok fails on wikipedia entry external links
Yep, robots.txt / user-agent exclusion is the problem
$mech->agent_alias( 'Windows IE 6' ); works with wikipedia but for some reason not gnu.org $mech->agent_alias('Linux Mozilla'); works for both.
I guess if wikipedia doesn't want mech scraping, I won't do it.
Thanks for your help planetscape,
|
---|
In Section
Seekers of Perl Wisdom