deprecated has asked for the wisdom of the Perl Monks concerning the following question:

Okay. Code snippet is here:
my %dummies; foreach (1..100) { srand; my $extra = int(rand(256)); warn "ralph" . $extra . "\n"; $dummies{$_} = dacts ( "ralph" . $extra, "myvoiceismypassword", "127.0.0.1", 8888 ); } for (;;) { }
this code is only slightly different from the code available in this node concerning writing to a slew of sockets with a threaded module, MP3::Napster. As you can (almost) see, this small codelet just spawns 100 new objects ($dummies{$_}), which all sit there and do nothing. There is actually an intended function for it, but as yet, I cant get to the 100 objects. What happens is the script starts up and spawns like crazy, very quickly, and then stops at right around 65 objects. Furthermore, most of the functionality on the computer ceases. So I ran it nice -n 20, and had a qx,top, runing at a -20 priority and I STILL couldnt see what perl was doing after about the 65th new object. At this point I decided I needed to break out the heavy artillery and ran strace. Eeep, this was alarming. The output of strace is here (at least the last 10 or so lines):

write(2, "connected to server\n", 20connected to server ) = 20 write(2, "username: ralph187 | password: m"..., 82username: ralph187 | + password: myvoiceismypassword at alternativejoin.pl line 17. ) = 82 rt_sigaction(SIGPIPE, {SIG_DFL}, {SIG_DFL}, 8) = 0 rt_sigaction(SIGPIPE, {SIG_IGN}, {SIG_DFL}, 8) = 0 write(67, "3\0\2\0 ralph187 myvoiceismypasswor"..., 55) = -1 EAGAIN (R +esource temporarily unavailable) rt_sigaction(SIGPIPE, {SIG_DFL}, {SIG_IGN}, 8) = 0 time(NULL) = 979853416 time(NULL) = 979853416 _newselect(0x48, 0x1061a1f0, 0, 0, 0x7ffff938 <unfinished ...>

I thought about this for a little while and decided to check /var/log/messages. I guess not very surprisingly, I found:

Jan 18 16:23:52 macachu kernel: __alloc_pages: 2-order allocation fail +ed. Jan 18 16:23:52 macachu last message repeated 639 times Jan 18 16:23:52 macachu kernel: ed. Jan 18 16:23:52 macachu kernel: __alloc_pages: 2-order allocation fail +ed. Jan 18 16:26:46 macachu last message repeated 455 times Jan 18 16:28:41 macachu last message repeated 123178 times Jan 18 16:28:55 macachu last message repeated 30800 times Jan 18 16:28:55 macachu kernel: ed.

Well duuhhhh, youre out of memory, deprecated! Not so fast. I have 192MB of physical memory and 384MB of swap. And neither disk was swapping. So while this seems to be a system limitation, it also doesnt seem to be using up all the memory available. I see some stuff in the strace about SIGPIPE and so on...? Should I be capturing it in perl? i.e., $SIG{PIPE} = sub { handle here }; ??

Relevant information:

Linux macachu 2.4.0-test11 #1 Thu Dec 7 15:51:21 EST 2000 ppc unknown cpu : 750 clock : 400MHz bogomips : 796.26 motherboard : PowerMac1,1 MacRISC Power Macintosh L2 cache : 1024K unified memory : 192MB
I've got to admit here, I'm pretty stumped. I would feel better about this if I appeared to be using all my ram.
Thanks fellow monks.

deprecated

--
i am not cool enough to have a signature.

Replies are listed 'Best First'.
Re (tilly) 1: So is this a kernel limitation, a hardware limitation, or a perl limitation?
by tilly (Archbishop) on Jan 19, 2001 at 03:44 UTC
    Well I don't have enough information to actually run that snippet. I have no idea what dacts() does.

    But my questions would start with, what does lsof say? Can you run the code on another machine which is not a test kernel, what does that do? (2.4.0 has a new VM, you may have hit a bad case for it. If you don't have another machine but are willing to post, you should be able to find someone who will run the experiment for you.) How much stack are you eating up?

    But while I don't have any suggestions for your problem, I have an unrelated suggestion. You should not be calling srand, Perl does that for you and has for several versions now. It is actually counterproductive to call it again. (On Linux you will be exhausting your system's entropy.)

Re: So is this a kernel limitation, a hardware limitation, or a perl limitation?
by Fastolfe (Vicar) on Jan 19, 2001 at 04:59 UTC
    Two things:
    1. The EAGAIN message is consistent with a write to an unavailable, non-blocking socket. It might have been placed into a non-blocking state before a connect call was issued and the script might be attempting to write to it before that connection was actually established. If the socket can't be written to yet, and you attempt to write to it, expect an EAGAIN "error" upon your write. This might just be due to a poor implementation of non-blocking IO.
    2. Instead of doing for (;;) { }, just sleep for a while. What you're basically doing is putting your script in a rather tight infinite loop, which will effectively hold your CPU usage at 100%. Most Unix kernels kind of figure that it's not doing anything productive and lower the priority of the process accordingly, so you might not notice any effects on other processes, but it would help diagnose the problem if your script was sleeping idly instead of working in a tight loop there. One less bit of activity on your system that might be catalyzing a separate problem.
Re: So is this a kernel limitation, a hardware limitation, or a perl limitation?
by AgentM (Curate) on Jan 19, 2001 at 04:09 UTC
    Whenever you compile a program under Un*x, you have four potential RAM limitations:
    • the compile-time RAM limitation
    • the so-called "soft" limitation
    • the hard limitation
    • the physical limitation
    It is entirely possible that you are using up the memory that was allowed for Perl to use and that Perl was compiled with a too-low-for-you mem limitation. If it is an option, you might try recompiling Perl with the -dm option to request what this compile-time limitation is. You say your system maxes out at 65 iterations. Why not fire up 10 iterations or even 1 iteration and first make sure that (1)Your code works as you request (2) and there are no memory leaks before fooling around with Perl's internals.

    Also, it looks to me like your program could benefit from threads if dacts() is forking off Napster clients. Unfortunately, threads are in pretty poor shape in Perl. Click here to see why. A C program with an embedded Perl interpreter might help you initialize and get you ready to run but the threads would heavily cut down on memory usage by those clients. If dacts() does not fork than you are confused because you use terms like "threaded" and "spawn" and thus make this paragraph null. Though, as tilly mentions, no one can imagine what dacts() does, if your system is coming to a halt, then I imagine that a large amount of processes are being forked. Another thing to ask yourself, also, is: Do I need so many Napster clients?

    AgentM Systems nor Nasca Enterprises nor Bone::Easy nor Macperl is responsible for the comments made by AgentM. Remember, you can build any logical system with NOR.
      By Popular Request, the dacts() function:
      sub dacts { my ($un, $pwd, $srvr, $port) = @_; my $r; my $n; warn qq-attempting $srvr:$port- if $DEBUG; $r = $n = MP3::Napster -> new ( qq-$srvr:$port- ); if ($r) { warn qq-connected to server\n- if $DEBUG } else { warn qq- +never connected to $srvr:$port- ; return } warn qq-username: $un | password: $pwd-; $n -> my_login ( $un, $pwd, qq-ralphnap-, LINK_UNKNOWN, 8899 ); if (!$n) { warn qq-could not log in to server $srvr : $! \n $@- ; +return } warn qq-logged in okay...\n- if $DEBUG; return $n; 1 }

      and, while i appreciate it, AgentM, I of course ran it with one instance, and it works. It is working fine as I write this, with 60 connections up (and netscape and an opennap server and zillions of rxvt's. When I say that it doesnt fork(), I mean that one process is running. It is my understanding from perldoc -f fork ...

      Does a fork(2) system call to create a new process running the same program at the same point. It returns the child pid to the parent process, C<0> to the child process, or C<undef> if the fork is unsuccessful. File descriptors (and sometimes locks on those descript +ors)
      actually spawns a new process. The amount of memory actually used in this case is pretty slim compared to what most would expect:
      MemTotal: 189248 kB MemFree: 7540 kB SwapTotal: 393156 kB SwapFree: 380144 kB
      So I think I have concluded here that memory is not the issue, unless something else is coming up that I'm missing.

      --
      i am not cool enough to have a signature.

        You are running out of handles somewhere. Since 65 is really close to a magic number (2**6 == 64) I'd bet that all those connections to the network are hitting a limit.

        --
        $you = new YOU;
        honk() if $you->love(perl)

      Hi, all. I just wanted to follow up on this post since it seemed to have a few people (or at least me) stumped. Well, I had to reboot the system for some maintenance, and I was then able to open over 2,000 connections with a modest 40MB of RAM. Clearly whatever the problem was, it was a software issue that was eluding me. All is well now. :)

      thanks to all who responded.
      dep

      --
      i am not cool enough to have a signature.