When you install LWP a couple of perl scripts are installed at the same time called lwp-rget and lwp-request. lwp-rget is quite a nice spider program supporting basic auth and cookie files so that may be what you're looking for.

---If it doesn't fit use a bigger hammer