BrowserUk: Sorry, but searching or parsing a large file, or scanning a directory tree are certainly *not* insignificant.

Large files and directory trees are not the general cases I meant by "generally insignificant".

BrowserUk: your suggestion that I might need to dump {lots of functions} and instead learn to use the aio_* varients from IO::AIO, along with all the entirely new and very different coding methods it requires: {some code} no matter how good that module may be, confirms my "forget everything else you know about writing Perl" statement exactly.

You seem to think I advocate IO::AIO for the general case. I don't. I clearly stated it was for high-performance programs that can't wait for the filesystem.

As I said elsewhere, your application can block for as long as you care to tolerate it. That can even be "indefinitely".

Threads introduce their own brand of complexity, and require their own "new" and "different" coding methods.

BrowserUk: Can I use {specific modules}?

I've used several of those modules successfully with POE. The ones I've never used can probably be made asynchronous in a separate thread or process. The exact method should depend on specific practical considerations, not the liberal immolation of straw men.

BrowserUk: Shipping large volumes of data to another process, and then reading the results back again is not efficient {....}

Threads don't really help here. Perl is forced to serialize and ship data between isolated interpreters, each in its own thread. This is pretty much the same pattern as serialized IPC.

Even worse, Perl's data copying is triggered by tie()-like magic. If your threads work on the same shared data, they're probably propagating all their changes to N-1 other threads.

Multiprocess designs are more future-resistant than multithreaded ones. When the application must scale beyond a single machine, the multithreaded design is scrapped while the multiprocess design is extended and redeployed.

BrowserUk: I'm am saying that it requires a programmer to learn an entirely different way of working that is much harder to learn, much harder to code and much harder to debug than the standard linear flow--do this, then do that, then do something else--that a single tasking program uses.

The same can be said about threads: Useful multithreading in Perl requires programmers to learn an entirely different, much harder way of working. Multithreaded Perl programs are a lot harder to write than their singlethreaded counterparts, and timing and data scoping issues make them way harder to debug.

BrowserUk: you cannot even run a simple sort on a moderately large dataset within a cooperative environment, because it cannot be interrupted. {...} {large-data argument removed because threads aren't much better; see above}

That's the problem with vague and poorly specified problems. Any "proposed solution" is going to be wrong once the details come to light. And if the details are as malleable as yours, the only viable solution will be the one you prefer.

For example, if we're talking about a highly-scalable data processing application, I might suggest using a database server rather than a disk file. This puts the data somewhere central, and a CPU farm can access it in parallel. If the database becomes your bottleneck, you can replicate it across multiple machines to divide the load.

The choice between callbacks or threads isn't pertinent at this level of design.

rcaputo: POE provides ways to maintain state between cooperative callbacks, but the POE-agnostic parts of a program don't even need that.

BrowserUk: You mean I can't just use lexical variables anymore? Isn't that a retrograde step?

Check out Lexical-Persistence if you want to store persistent POE state in lexical variables. The eg directory includes a simple POE example. If you need a higher level of abstraction, you're invited to write one.

Also keep an eye on POE-Stage. It's the project that spun off Lexical::Persistence, and it makes heavy use of persistent lexical variables.

rcaputo: A program that requires major restructuring to work in a cooperative environment may already have bigger problems.

BrowserUk: That sounds a lot like 'if your double buggy doesn't fit through my turnstill; it's your double buggy that's a fault'!

My turnstile is not the doubly buggy thing here. But seriously, to reiterate and clarify my point: Well-designed code tends to be easier to make cooperative than monolithic messes. And if you have a monolithic mess, you have other issues.

BrowserUk: Most (Perl) users can't (or don't want to and shouldn't have to; and for most programming problems, don't have to), fabricate their own (POE) bricks. Much less have to re-fabricate other peoples existing, working, tested, freely availble open-source software bricks. Again, *not* bogus. And with threads, they don't have to.

Are you claiming that most Perl programmers are incompetent or lazy? Woo-woo! Well, laziness is a virtue, but this smells like the dreaded false laziness.

Oh, and threads work with POE, so everything works with POE by the transitive property of "threads make everything okay". :)

By the way, check out SEDA. Matt Walsh has shown that an amalgam of threads and asynchronous I/O performs better than either by itself. SEDA stands for "Staged Event Driven Architecture". It's the inspiration for POE::Stage's name.

Thanks for reading.