slow, slow IO::Select

dash2 has asked for the wisdom of the Perl Monks concerning the following question:

I'm trying to speed up communication to a bunch of WHOIS servers by using IO::Select - the theory being that it is better to make 100 queries and then sit around for the results, than to make each query one by one and wait for each result sequentially. But instead of speeding up, my code is timing out the web server! I must be doing something dumb, but I can't figure out what.

Here's the stuff:

# create a nice  IO::Select object with a bunch of sockets... then
while ( my @ready = $select->can_read(30) ) {
  foreach (@ready) {
   my $result = sysread($_, $domains{$_}[2], 128$domains{$_}[3]); # do
+mains is just a local data structure where elements [2] and [3] are t
+he WHOIS results returned so far, and the offset
   unless (defined $result){
    $domains{$_}[0]{result} = "no_connect";
    $select -> remove($_);
    $_->close;
    next;
   };
    if ( $domains{$_}[2] =~ /$domains{$_}[1]/  ) {
    $domains{$_}[0]{result} = "avail";
    $select -> remove($_);
    $_->close;
    next;
   }
   unless ($result) {
    $domains{$_}[0]{result} = "unav";
    $select -> remove($_);
    $_->close;
    next;
   }

   $domains{$_}[3] += $result;
  }

  foreach ($select -> has_exception ) {
   $select->remove($_);
   $_->close;
  };
 }
[download]

Am I being stupid in mixing sysreads with IO::Selects? Or not checking for something obvious?

Thanks

dave hj~

Comment on slow, slow IO::Select Download Code

Replies are listed 'Best First'.
Re: slow, slow IO::Select by dash2 (Hermit) on Mar 13, 2001 at 06:29 UTC
duh, that'll be like a yes - the has_exception call is blocking, because it is without a timeout. I don't really want to wait a second every time I go through the while loop, just to check for exceptions, so I think I'll kill that bit. I didn't really understand it anyway... what is an "exception" and what is "out of band data"? Hmm. Must understand more about socket programming. I haven't yet figured out how to optimise this thing... wonder what the best chunk size to get data is from the whois server? dave hj~	[reply]
Re: Re: slow, slow IO::Select by AgentM (Curate) on Mar 13, 2001 at 06:38 UTC
Exceptions come around in Perl usually in the form of signals. Generally, Perl code return "bad" values from functions instead of generating "exceptions". (Exceptions come around in C++ and Java among others.) OOB (out of band data) generally generates a signal (unless you turn that off) and is placed in a high-part of the TCP/UDP data being sent. It was designed for "emergency" information but it remains generally unused in everyday apps. It can be used for anything you wish if you know how to throw some OOB data together, send it, and capture it. Otherwise, you get SIGURG (urgent) and bomb. Do the RIPE/WHOIS servers actually use that? I wasn't aware of that. M$ History Lesson of the Day: A few years back, it was discovered that many ports on a Windows machine choked on OOB data, causing anything from a full system lockup to a BSOD. Hence, winnuke was born, which was essentially 3 lines of real code. Thanks, M$! AgentM Systems nor Nasca Enterprises nor Bone::Easy nor Macperl is responsible for the comments made by AgentM. Remember, you can build any logical system with NOR.	[reply]
(tye)Re: slow, slow IO::Select by tye (Sage) on Mar 13, 2001 at 21:11 UTC
I'll disagree with AgentM on exceptions. With select, an "exception" means an error, for example, the other side resetting the connection. "Has exceptions" roughly means that a read or write attempt would fail rather than hang while "can read" and "can write" means that such calls would succeed rather than hang. I'll agree with him about OOB data. (: - tye (but my friends call me "Tye")	[reply]

AgentM Systems nor Nasca Enterprises nor Bone::Easy nor Macperl is responsible for the comments made by AgentM. Remember, you can build any logical system with NOR.