in reply to Subdir globs

Take a look at wget, located at Excerpt from that page:

The recursive retrieval of HTML pages, as well as FTP sites, is supported: you can use Wget to make mirrors of archives and home pages, or traverse the web like a WWW robot (Wget understands /robots.txt)