I would use Threads::Queue to send messages back from your threads to your main thread instead of polling them for becoming ->joinable.
For passing thread results back otherwise, see threads, which documents the return value of ->join(). I would avoid using that, because to get more speed, you will want to keep a pool of worker threads and pass the queries to them using another Thread::Queue instead, thus saving the cost of creating and destroying a new thread for every request.
| [reply]
[d/l]
[select] |
*Banging head against table*
Thanks. I was searching for that precisely: sending back messages. Saw Threads::Queue yesterday and the words "Queue" dissuaded me from looking into the documentation.
| [reply] |
Still, in light of BrowserUks discussion, you should consider what the "main" thread is doing at all. I think that most of the work per-client can be done in a worker threads. Instead of just farming out the HTTP requests, I would farm out sending the responses to worker threads as well.
| [reply] |
Hm. Why do you want to get the results back in the main thread? What are you going to do with them there, that you couldn't do with it in the thread where you obtained them?
You've set up your main thread to read (follow?) this logfile. That's a nicely defined, and self regulating division of labour. What makes it necessary to confuse that by requiring it to also check for and do something with the results?
Any information that is available to the "main" thread (which is just another thread after all), is (or can be) available to each of the checking threads you spawn. So what stops them being able to complete the processing required for the MAC they were given?
Of course, there are occasions where it is required, or just easier, to perform some final processing on the results from different threads in a common place. But there is still no need to confuse the log reading thread by trying to multiplex this workload in there. Simply start another thread to take care of it.
Or, if it makes sense that this final processing take place in the "main" thread, spawn another thread at the beginning of the program to do the reading from the file. That way your "main" thread can concentrate upon the final processing.
And so I come back to the same questions I asked earlier. Can you post a high level overview of the processing requirements of the application? It would make it much easier to suggest a good solution.
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
| [reply] |
What the working non-threaded stuff does is this:
Looks for DHCPACK in logs.
Parses MAC from it.
Checks with a http server which responds with a pass/fail.
If http response is success, sends a RADIUS packet for Authentication.
Apart from this, I need to monitor MACs that have failed and also those that have passed continuously.
What I am worried about in my non-threaded model is that when several DHCPACKs come at the same time and one request takes a little longer than expected, it will block all the other MACS in queue => Hence threads to do the API and RADIUS checks.
As mentioned in my earlier post , pumping 10 DHCP acks into the log files saw 10 threads being created simultaneously. BTW, not implemented the radius part yet. ONly the pass/fail part is being tried.
Each response takes roughly 1 second in the network normally. The first thread does that but then it progressively worsens . The 10th takes 8 seconds. The reason I *think* is how the threads are scheduled and the join. From what I have read it looks like join blocks the other threads. So if you look at the 10th thread, it is blocked by the 9 threads before by their joins.
I even tried detach instead of join. Again, it progressively worsens as it reaches the 10th thread. I dont know why.
I am looking at ways to for each http call to take roughly the same time and finish and be able to have a hash in the main thread w the status for each MAC I have encountered.
| [reply] |
I even tried detach instead of join. Again, it progressively worsens as it reaches the 10th thread. I dont know why.
Can you let me see that code? I might then be able to answer the why.
From what I have read it looks like join blocks the other threads
Where did you read that? join() only blocks the thread is runs in.
I suspect your misunderstand arises from $_->join for @threads.
If the first thread in @threads is last to finish, then the other threads will not be joined until that occurs. But that is because you've asked to join the threads in a specific order. So don't do that.
Again, sight of your code would make life simpler.
and be able to have a hash in the main thread w the status for each MAC I have encountered.
This is the bit I'm not understanding yet.
- Why must the main thread retain this information?
- What will it do with that information?
- When will it do it?
By way of example. This simulates reading a logfile (it just reads from STDIN); spawns a thread for compliant input (your ack lines); those threads "process" the input (sleep and then build a hash for return); and a finisher thread processes the results (prints them to stdout).
The thing to note is the separation of concerns. The main thread only reads the log; the spawned threads only deal with their job and return results; the finisher thread gathers and processes the results. Each thread is free to do what it needs to do, when it need to do it, without any form of synchronisation or locking.
Each thread individually is a simple linear piece of code. It either starts, runs and finishes; or loops until done. No multiplexing; no synchronisation; no artificial events; no breaking up linear code into itsy bitsy chunks in order to ensure that someother concern is dealt with in a timely manner. We let the OS scheduler ensure that when a trhead needs to do something, it gets done just as soon as there is a cpu available to do it. That means that it runs just as well (actually better) on a 16 or 32 core processor as it does on a 2 core processor, without the need for any hardware specific tuning. It just works.
#! perl -slw
use strict;
use Time::HiRes qw[ sleep ];
use Data::Dump qw[ pp ];
use threads;
use threads::shared;
my $done :shared = 0;
sub checkStuff {
sleep rand( 5 );
return { split ' ', $_[0] };
}
my $finisher = async {
while( sleep 0.1 and not $done ) {
for my $thr ( threads->list( threads::joinable ) ) {
my $hRef = $thr->join;
printf "From tid:%d got %s\n", $thr->tid, pp $hRef;
}
}
};
while( <> ) {
next unless /ACK (.+)$/;
async \&checkStuff, $1;
}
sleep 0.1 while threads->list( threads::running ) > 1;
sleep 0.1 while threads->list( threads::joinable );
$done = 1;
$finisher->join;
__END__
[15:14:01.92] c:\test>856740-2
ACK A 1 B 2 C 3
junk
ACK A 1 B 2 C 3
ACK A 1 B 2 C 3From tid:2 got { A => 1, B => 2, C => 3 }
junk
ACK A 1 B 2 C 3
junk
junkFrom tid:3 got { A => 1, B => 2, C => 3 }
ACK A 1 B 2 C 3From tid:4 got { A => 1, B => 2, C => 3 }
junk
^Z
From tid:5 got { A => 1, B => 2, C => 3 }
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
| [reply]
[d/l]
[select] |