in reply to Re: Re: Simultaneous writes and as-needed reads from sockets (or The State of Perl Threads...)
in thread Simultaneous writes and as-needed reads from sockets (or The State of Perl Threads...)

and just send a login request to all of them, and then deal with them as they get back to me, or, if necessary, destroy the connection. Instead, it takes 10 minutes to get to only 60 of the servers because I am waiting for every one of them.

This doesn't sound like select-based behavior. Perhaps you're mixing select with standard blocking calls? A good non-blocking select-based implementation of something like this should be able to handle dozens of network sockets simultaneously without a significant amount of delay. If you're hitting multiple servers, you should be able to number your simultaneous connections in the hundreds and under any decent system, your bottleneck will be with your network connection, not the app (unless you're doing a lot of processing with the inbound data I guess).

I mean, there are two ways you can go about this. I have no idea how Napster.pm does its thing. If you say it's working with select, fine. I don't get what the purpose of multiple threads is, in that case, but whatever. It's not important. With select, you work with Perl filehandles. These can be network sockets, files, STDIN/STDOUT, pipes, whatever. If you want to use open($S, "-|") to fork off a child process, IO::Select would be happy to use $S as a filehandle to watch. You can repeat this a dozen times to get the behavior you're looking for, with each child doing an exec or whatever it is you want. The select call will be happy to tell you which file handles have data waiting to be read.

So basically, going back to your problem, it sounds like you neither want nor can code any direct hooks into the way this other module is doing its network handling. So realistically you can't make it "go faster" when it's working with multiple servers. What, then, do you plan to do? Do you want to fork off your process into 40 sub-processes, each one devoted to a single server? If so, select is still very much an option. Instead of selecting against network sockets, select against a pipes (such as that perlipc version of open above), and process inbound data from each of your children in turn.

If something like this is the route you want to take, I highly recommend reading perlipc. You can always just fork and let each child write to STDOUT, but it's difficult to *capture* that information in a controlled way without using true pipes and mediating between them by using select, so that you can avoid blocking while waiting for data from one of them.

If I've missed the boat on this, if your plan to break these tasks up is altogether more bizarre than anything I've mentioned, by all means let me know.

  • Comment on Re: Re: Re: Simultaneous writes and as-needed reads from sockets (or The State of Perl Threads...)
  • Download Code