in reply to Re^2: Performance and CPU load: sysread, IO::Select and/or Storable::thaw
in thread Performance and CPU load: sysread, IO::Select and/or Storable::thaw

I would have uncommented any calls to $handle->blocking(0). The default mode is blocking, so that should work, and work even after forks.

Naturally that could make other problems if some other part of the code relies on non-blocking reads, but it would be only for testing the hypothesis

The timeout of 5 seconds is quite ok. If the loop only runs every 5 seconds you will never notice it. What the docs mean is that if you don't specify a TIMEOUT value, the timeout value will be infinity. But 5 seconds is practically infinity, if you look at cpu load

So uncomment or remove the blocking(0) calls, add print lines around your can_read calls to check if they wait for 5 seconds. If they still don't wait, check how many bytes they read out of the socket, maybe whatever code that writes to the socket is broken and spams the socket with data.

  • Comment on Re^3: Performance and CPU load: sysread, IO::Select and/or Storable::thaw
  • Download Code

Replies are listed 'Best First'.
Re^4: Performance and CPU load: sysread, IO::Select and/or Storable::thaw
by DBX (Pilgrim) on Jun 29, 2010 at 19:42 UTC

    In theory, your comments make perfect sense and your advice is great. I've actually set up a full test environment for these and tried every variation of blocking() including commenting it out as you suggest. I've also tried replacing can_read() with IO::Select::select().

    In every single case, the IO::Handle object returns 1 for blocking(), but can_read returns immediately, regardless of timeout value. I am almost 100% certain the parent is not sending any more data than the message that needs to be thawed. No matter what, CPU spikes to 100% in the child until it dies. The only conditions that mitigate this are when I reduce the timeout on can_read() or select and even then CPU is between 60-80% for each child. Improved, but still a resource drain.

    I'm very perplexed as to why can_read() and select will not block in this application.

      I looked a bit closer at your code and noticed that you add STDIN to your select call instead of readHandle. Any reason? It might be that this is your problem.

      I tried the following test script and it works. I get output every 5 seconds:

      #!/usr/bin/perl use warnings; use Data::Dumper; use IO::Handle; use IPC::Open2; use IO::Select; my $i; my ($readHandle, $writeHandle) = (IO::Handle->new, IO::Handle->new); my $pid = open2($readHandle, $writeHandle, 'bc'); $writeHandle->autoflush(1); $readHandle->autoflush(1); my $select = IO::Select->new(); $select->add($readHandle); while (1) { print $i++,"\n"; my ($handle) = $select->can_read(5); }

      Interestingly it works even when I turn off blocking, which means my previous notion that can_read would not wait if blocking is turned off might be false.

        The $readHandle is defined in the parent, not the child. The child is reading from STDIN, so I don't believe this will work, and in my testing, it did not.

        I do, however, have a workaround. I set the timeout to 0 based on your advice and put a very small delay at the end of the "infinite" loop:

        while(time < $endTime) { [...] select(undef, undef, undef, 0.05); }

        Since I could not get blocking to work no matter what I tried (which may still be some fault of my own), this seems to be a workable fix for now. It reduced CPU to almost nothing, which is an enormous gain since the load was consistently over 80 on the box where this was running multiple instances.

        I will continue to modify my test code to see if I can implement your suggestions more effectively. Your help has been invaluable in at least finding a workaround and possibly a permanent solution, jethro, and I really appreciate your time