I do lots of scraping/automation with LWP/WWW::Mechanize, I found the following two are very useful:

• Ethereal, so you can see what actually gets sent, and compare that with what the browser is sending.
• "perl -d", yes, perl debugger, stepping through the code is really useful.

I am able to solve most of my problems using these two (except those with heavy javascript.)