That's not the issue, minicpan, unless you're doing something weird, should only pull back the latest releases required to build distributions. Over the years people have uploaded many modules, and some very large in the App space (including vast bundles of other software). Unless you configure it to ignore bloat then you won't avoid this, and even then I've come across legitimate modules that have a dependency on ACME modules (for 'test' data).

How do you define bloat in a filterable way?

(Did I miss a bloat flag in the meta files? ;)

FWIW: For the purpose of this thread downloading only pure text like Perl code should be fine. (or just excluding any binary)

Cheers Rolf
_{(addicted to the Perl Programming Language :)
Wikisyntax for the Monastery}

I made a mirror, was shocked at the size, then dug into where all the disk space was going. Stuff I'd never use, single App distros that were >100mb, apps for platforms I'd never use, ridiculous, insecure crap nobody should use.

> Unless you configure it to ignore bloat then you won't avoid this

For the aim of parsing all Perl&POD source locally I'd need to pull all text and ignore binaries and other "bloat" (to be defined) to save disc space.

But this won't be faster in net-load, since AFAIK does filtering happen after downloading the full dist's tgz. °

Cheers Rolf
_{(addicted to the Perl Programming Language :)
Wikisyntax for the Monastery}

°) well probably avoiding extracting certain files from the tgz might speed up things a little tho.

You remember correctly. However, some authors are unreachable and/or apathetic and the number of dists constantly increases as well.