Re: a IO:SELECT problem

Interesting question. My first thought is: What's the big deal about 200 processes or threads? That would almost certainly be the easiest solution.

Assuming there's some good reason why that won't work, what you would need to be able to do is asynchronous database operations. Either a way to get a callback or a signal when results are ready, an implementation that guaranteed that some filehandle would be selectable only when there's results to read, or at least something you could poll with. You could look at your database's driver options to see if this is possible; if only the C version of the driver supports it, you could use Inline::C to talk to it.

Another option would be to proxy the results through something that would send you some kind of notification, or only make a filehandle selectable when results were ready. You could use some kind of RPC interface, like SOAP, or extend something like DBD::pNet to give it these capabilities.

But those are much harder than managing 200 threads or processes. :-)

--
sgifford's Web page

Comment on Re: a IO:SELECT problem Select or Download Code

Replies are listed 'Best First'.
Re^2: a IO:SELECT problem by ablmf (Acolyte) on Jun 26, 2006 at 14:50 UTC
Thanks a lot for all your help. But won't using 200 processes or thread be too resource consumable comparing to use select or poll? If it would considerable low down the performance of the program, it would be unacceptable. My program will running on a linux server. By the way, I am using mysql. Does it provide asynchronous operation?	[reply]
Re^3: a IO:SELECT problem by sgifford (Prior) on Jun 26, 2006 at 15:39 UTC
It depends on what the processes are doing; if they're mostly sleeping while waiting for results to become available, resource use will be minimal. If they're all trying to do a lot of work at the same time, they may overwhelm the CPU, and you'll have to be careful to throttle them a bit. Parallel::ForkManager can be helpful with that sort of thing. I just glanced through the MySQL docs, and I didn't see any kind of asynchronous API. You may get some help on the MySQL mailing list; there might be a clever hack that will allow this. -- sgifford's Web page	[reply]
Re^3: a IO:SELECT problem by BrowserUk (Patriarch) on Jun 26, 2006 at 15:52 UTC
Unless your MySQl server is running distributed, or on a machine that has dozens of processors, your 200 'concurrent' queries are going to be seriealised anyway. It might be better to start a (small, 2 or 3) pool of DB worker threads in your application, and queue the queiries and results to/from the server. However you choose to do it, only 1 or 2 of your 200 queries are actually going to run concurrently, and the 200 threads/processes that issue those queries will all be waiting for their results anyway. It's really quite hard to see what you hope to achieve by issuing concurrent queries from a client process? Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal? "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply]
Re^4: a IO:SELECT problem by ablmf (Acolyte) on Jun 26, 2006 at 16:09 UTC
Do you mean that, the best way to solve this problem, is to have all the most frequently used data kept in the memory? Maybe having concurrent queries won't save the total time of query, but at least I could avoid that my server could not do anything when waiting for query result. For example, the server might process some request that do not need DB queries while other connections are waiting for DB response.	[reply]