What I'm trying to do.. A service which will do "stuff" (it's a generic architecture to be customised to particular applications, so I really mean stuff) to bundles of streams - essentially, a client will tell it "here's a bunch of input streams, and a corresponding bunch of output streams, process 'em using some-mechanism-or-other as you pump data from the inputs to the outputs". Each group of inputs and outputs are collated in some way for purposes of error correction, and the service will process multiple bundles simultaneously. Little example diagram here, with two bundles: the first (A->B) is unidirectional with three inputs and only two outputs, while the second (C<->D) is bidirectional with one stream at one end and three at the other (the actual stuff that multiplexes/demultiplexes doesn't really matter for this question).
A0--\ /B0 A1---+->stuff>--+ Bundle 0 A2--/ \B1 /--D0 C0---<stuff>--+---D1 Bundle 1 \--D2 ... Bundle N ...
I'm using Perl 'cos it facilitates rapid prototyping (this is research work), it has good network handling, and is easy to integrate with CGI, web services, and third-party applications (all of which are desirable in this case).
The architecture I've gone with, due to the relatively heavy weight of Perl's threads, is to set up a single worker thread which processes all of the bundles in a select() loop, and then have the main thread obtain, validate and submit control directives (such as adding new bundles or modifying existing ones). In an ideal world, I'd just turn each control directive into a closure and hand it off to the worker thread to deal with in its own good time. In another language that might be easy while all the other stuff I've mentioned might be hard; in Perl, the other stuff is easy and this, while not hard to hack (I'm already doing it by passing filenos across) seems hard to do nicely.
The reason for separating the control invocation from the worker thread is that I envisage a variety of different control styles, e.g. web page, web service interface, and straight program API. I'd rather not get that all munged up with the I/O and processing work...
So I escape your wise warning re: freezing objects because the control thread destroys its copies of them as soon as they're frozen and passed off to the worker. It's clearly not efficient, but my purpose for the time being is clarity and flexibility, which whole-object transmission gives me. Passing instructions through as unblessed shared string/list/hash structures was more error-prone.
Now on to the file descriptor messaging: at the moment I do one of two things: either I pass across coordinates (such as "hostname:port") and the worker initiates a socket connection itself, or I pass across the file descriptor of a socket/pipe/filehandle and the worker tries to reconstruct the original object using IO::xxx->new_from_fd(). What would be nice would be to be able to just pass across IO::xxx objects; since that's not possible without extending Storable (I am tempted, would make everything tidy again), the necessary thing seems to be to deconstruct the objects into class (necessary since I can usefully half-close socket connections but not files for example) and file descriptor. Your open("&=") trick is analogous to the new_from_fd one, and difficult because it too needs to know the opening mode of the original object (you have to specify "<&=", ">>&=", etc.). So I guess my ultimate questions are:
In reply to Re^4: Passing globs between threads
by conrad
in thread Passing globs between threads
by conrad
| For: | Use: | ||
| & | & | ||
| < | < | ||
| > | > | ||
| [ | [ | ||
| ] | ] |