(Did I miss a bloat flag in the meta files? ;)

FWIW: For the purpose of this thread downloading only pure text like Perl code should be fine. (or just excluding any binary)

I made a mirror, was shocked at the size, then dug into where all the disk space was going. Stuff I'd never use, single App distros that were >100mb, apps for platforms I'd never use, ridiculous, insecure crap nobody should use.

I followed your link out of curiosity, and FullAuto doesn’t seem bad to me. Orchestrating dozens of servers can be a pain. I usually just use .ssh/authorized_keys but it does have a connection overhead on every single remote command I run. I get around that by putting scripts on the remote host so that I’m only making a few ssh calls and the remote scripts do the lifting. FullAuto looks like a reasonable alternative design.

so you are manually maintaining a blacklist of distros?

Cheers Rolf
_{(addicted to the Perl Programming Language :)
Wikisyntax for the Monastery}

For the aim of parsing all Perl&POD source locally I'd need to pull all text and ignore binaries and other "bloat" (to be defined) to save disc space.

But this won't be faster in net-load, since AFAIK does filtering happen after downloading the full dist's tgz. °

°) well probably avoiding extracting certain files from the tgz might speed up things a little tho.