Re^4: Win32 - flock() not FIFO?

BrowserUK,

Threads / shared / queues opens up a lot of possibilities that will really simplify my application. Already I see how one of my select(...,timeout)'s will go away. The other one needs a little more thought, but I think it can be restructured along the lines you suggest.

Under the fork-exec model, the five processes start with static copies of the same variables. A change to any variable requires propagating it to the other processes. This is done with udp datagrams. Each of the four worker process talks only to the "supervisor", who relays the messages (see below).

There are three basic messages passed between processes:
Request: ("get", $item, "") or ("set", $item, $state)
Response: ( "status", $item, $state)

Four of the processes do actual work, while the "supervisor" primarily just relays messages. The get and set messages are one-to-one, but the status message must be cc'd to the other processes so everyone knows the current status of $item. To cut down on the number of messages, the supervisor doesn't cc anyone if the status has not changed. The supervisor also provides the "watchdog" function, waking up every 60 seconds. An external digital i/o PLC will do an "automated scram" if it has not received a command from the software in 120 seconds, so the watchdog function keeps it running. (Yes, the operator can initiate a "manual scram" too.) The watchdog also requests the status of any physical hardware device if no one else has asked for it in the last 60 seconds.

Enter threads / shared / queues: If I make %status a shared variable, then all the status response messages go away, since each thread will have access to the current $status of any item. Supervisor's job is now reduced to just doing the watchdog function: sleep(60); do_watchdog(); No select needed.

The gets and sets are unidirectional one-to-one messages, so each worker thread can have an incoming queue which it monitors with Q->dequeue. A thread can issue a get or set request directly to the receiving thread by doing a Q->enqueue. Now all the udp messages go away.

The other select(...,timeout) process needs some more thought. It must asynchronously process get and set requests, while waking up every second to implement the PID (proportional, integral, derivative) ramp-and-soak algorithm, and turn Solid State Relays on and off. As you suggest, waking up every 1/10 second to check the Q will not consume much CPU, and 1/10 sec is probably asynchronous enough.

Question: Do the shared variables and queues need locking?

With a single CPU, I suspect the shared variable updates are atomic, but the Q operations might not be. With multiple or multi-core CPUs, things could possibly happen at exactly the same time, so problems might occur. Thoughts?

If it ain't broke, don't fix it. Well, I've already converted to threads, the system works perfectly and consumes 0% CPU. Implementing shared variables and queues will take some work, but the benefit would be easier maintenance - there's a lot of complicated logic that would simply go away.

I've got some other questions that I'll post in new messages.

AH HA MOMENT: I got really tired of typing in all the P and /P tags to make my PerlMonks posts look pretty. Then I realized that if I composed them in Dreamweaver and typed in the "display" area, DW would make a new paragraph every time I hit the enter key. Much easier!

Dave
The Electronic Brewery

Comment on Re^4: Win32 - flock() not FIFO?

Replies are listed 'Best First'.
Re^5: Win32 - flock() not FIFO? by BrowserUk (Patriarch) on Oct 30, 2011 at 17:05 UTC
If I make %status a shared variable, then all the status response messages go away, Now you're cookin' on gas :) Question: Do the shared variables and queues need locking? Shared variables do. Queues don't. (Or rather, they do as they are actually just shared arrays under the covers, but all the required locking is taken care of under those same covers, so you do not need to do it yourself.) With regard to atomicity on a single-cpu system. Do not rely on it. Applying locks is very easy and not worth skipping for anything more important than trivial experiments. there's a lot of complicated logic that would simply go away. Indeed. The benefits of shared state and linear code flows. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply]
Re^6: Win32 - flock() not FIFO? by hennesse (Beadle) on Nov 08, 2011 at 02:28 UTC
BrowserUK, I just wanted to get back to you and say THANK YOU for all the education you've given me on threads, shared variables, and thread queues. I followed the plan I laid out, said goodbye to all the datagrams and implemented all my IPC through shared variables and thread queues. There were a few speed bumps, but no major damage, and everything came out fine in the end. After that, I revisited my webserving strategy. The Electronic Brewery's operator console is a web page that loads once, and then refreshes its content every second via AJAX calls to the daemon (now a win32 service). I had waffled between three approaches (a) doing all the serving myself, (b) having Apache doing everything, with the AJAXes hitting a CGI, and (c) splitting the duties between me and Apache. (a) worked fine, but didn't have all the security stuff that Apache has already figured out. (b) worked, but Apache needed to spawn a perl.exe to handle an interface function that did almost nothing - at great expense. so (c) seemed the best compromise. Apache has figured out all the security for file serving, so it's the right tool for the job for loading the web page and all its appurtanances. The AJAXes are POSTs, url ignored, with with one line of content. So writing, securing, and handling error conditions for an AF_INET SOCK_STREAM server was not that bad. But once you've found a new toy, you want to use it for everything. Remember Marshall T. Rose's observation that "if the only tool you have is a screwdriver, then everything begins to look like a screw" ? Implementing a concurrent webserver with threads took a lot of time - it took .09 sec to create a new thread, but only .04 sec to service the AJAX request. So I ended up re-writing (doing a lot of that lately) the concurrent server as an iterative server. Now I'm over on the Javascript side, trying to clean up my works fine, but horribly inefficent, code Thanks again BrowserUK, you're a wealth of knowledge and a great teacher. Dave	[reply]
Re^7: Win32 - flock() not FIFO? by BrowserUk (Patriarch) on Nov 08, 2011 at 03:14 UTC
Implementing a concurrent webserver with threads took a lot of time - it took .09 sec to create a new thread, but only .04 sec to service the AJAX request. So I ended up re-writing (doing a lot of that lately) the concurrent server as an iterative server. I assume you were starting a new thread for each client? Perl's threading implementation carries to much overhead with starting a new thread to make that a viable option. Actually, I'd say that even in C that starting a new (kernel) thread for every brief exchange is an extremely wasteful way of handling connectionless protocols. Far better is to use a pool of threads. For an example of handling concurrent sockets see Re^7: multithreaded tcp listener with IO::Socket. It probably needs checking as it was written back in the 5.8.6 days and quite a lot has changed in the interim. If you are thinking of using it as the basis for anything, let me know and I'll give it the once over. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply]