Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change
 
PerlMonks  

CPAN module installation makes no progress on limited resource machine

by runderwo (Initiate)
on Apr 16, 2006 at 23:44 UTC ( [id://543730]=perlquestion: print w/replies, xml ) Need Help??

runderwo has asked for the wisdom of the Perl Monks concerning the following question:

This is an odd one (for me, a perl admin/user/developer of 5 years).

A friend has a fleet of P166MMX, 32MB laptops he is attempting to turn into digital photo frames. We installed Debian etch pre-release (with kernel 2.6.15, Perl 5.8.8) on one to use as a prototype. Everything went well, I did various things to keep the memory usage down such as removing excess gettys, replacing bash, etc ... and then we hit this stupid snag.

I have tried both the original CPAN as well as CPAN Plus. Both of them do the same thing. When I attempt to install a module through CPAN, at first things seem to be going ok, then it hits the part where it is building the module tree (before it actually fetches and installs the module). At this point, the system becomes unusable. CPAN eventually responds to a Ctrl-C, but it takes several seconds, and the process cleanup takes several minutes unless sent SIGKILL.

I have examined the process while this is happening. Top shows that the perl process is continuing to allocate memory, and spending time alternating between the R and D states. At no time is the perl process actually consuming 100% CPU or anywhere near it. An examination with strace shows that all that is happening is that time(NULL) is called hundreds of times per second, and brk() is occasionally called, always with a larger argument than the previous brk().

Based on this, it seems like the perl is stuck in an infinite memory allocation loop. But I don't know if it is really infinite. The process grows to an 18MB resident size, everything else is paged out, and the VIRT size keeps growing (last one I saw was around 80MB), until eventually my ssh session is paged out for too long and the connection dies. (He lives far enough away where doing this in person doesn't make sense.)

I have attempted to set /proc/sys/vm/overcommit_memory to 1 with the same results. Does anyone else have a better idea what is going on here? Does this perhaps have to do with Debian's perl using internal malloc rather than system malloc? Is there some kind of race going on due to the limited resources of the machine?

Running perl on some test scripts seems ok, but I'm not sure what a good test case for this is.

I used debian.pkgs.cpan.org for most of the stuff he needs, but on the other stuff it isn't packaged and has lots of dependencies which I'd prefer not to have to hand install...

  • Comment on CPAN module installation makes no progress on limited resource machine

Replies are listed 'Best First'.
Re: CPAN module installation makes no progress on limited resource machine
by PodMaster (Abbot) on Apr 17, 2006 at 07:56 UTC
    When I start cpan (win32, perl5.8.4), perl consumes about 12MB.
    When I issue "m Tie::PureDB", memory consumption climbs to 56MB.
    If I issue "reload index", memory consumption climbs to 80MB.
    Expanding
    04/17/2006 12:44 AM 113,646 01mailrc.txt.gz 04/17/2006 12:47 AM 437,330 02packages.details.txt.gz 04/17/2006 12:50 AM 114,181 03modlist.data.gz ================================== 04/16/2006 03:35 PM 283,775 01mailrc.txt 04/16/2006 10:58 AM 3,003,067 02packages.details.txt 04/16/2006 07:04 PM 396,897 03modlist.data
    and creating a giant data structure out of all this data, can consume quite a bit of memory.

    You need to examine the following (and probably other) CPAN settings (and their CPANPLUS equivalents, if any): build_cache, cache_metadata, dontload_hash.

    update: Looks like my loose sugestion on the CPAN settings probably isn't very useful :(
    #4943: CPAN module too memory intensive

    MJD says "you can't just make shit up and expect the computer to know what you mean, retardo!"
    I run a Win32 PPM repository for perl 5.6.x and 5.8.x -- I take requests (README).
    ** The third rule of perl club is a statement of fact: pod is sexy.

      Holy cow, thanks for the link. I tried searching for a while both on google and google groups and came up with nothing, so that's why I posted here. It's too bad that CPAN Plus did not address this problem. I am glad that I am not the only one with this problem though (i.e., I'm not crazy!)

      The reason I would prefer to use CPAN than manually installing all the packages is because tracking and manually downloading all the dependencies (and their dependencies) is a complete pain. PEAR for PHP works, after all, so should CPAN. ;) Are there any alternatives to CPAN/CPANplus for perl package management?

Re: CPAN module installation makes no progress on limited resource machine
by osunderdog (Deacon) on Apr 17, 2006 at 12:29 UTC

    I guess the only advice I would give is to not use CPAN. You can pull perl packages manually and build them yourself. The steps are pretty simple:

    • untar package
    • perl Makefile.PL
    • make test
    • make install

    Hazah! I'm Employed! Although still interested in better opportunities...

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://543730]
Approved by McDarren
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others having a coffee break in the Monastery: (4)
As of 2024-03-29 15:59 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found