Greetings,

Re: Is it handled by an MCE server process with it's own buffering and file I/O will be done on the server process and the result returned via IPC to the hobo?

Yes. The output-class constructs a shared object. Initially, the file handle ($fh) is not defined. This allows MCE::Shared->share to pass the object to the shared-manager, where it will run. The entry point to the shared-manager is the shared-object itself. Method calling and arguments are sent via IPC. The mechanism is fully wantarray aware. Any complex data during transfers involves serialization which is transparent to the application.

Regarding IO position tracking, there is none for shared file-handles constructed via MCE::Shared. The close method was added to the class, so that any pending writes, held in a buffer by Perl, are flushed to disk before exiting; e.g. $iter_o->close().

package Iter::Output;

sub new {
   my ( $class, $path ) = @_;
   my ( $order_id, $fh, %hold ) = ( 1 );

   # Note: Do not open the file handle here, during construction.
   # The reason is that sharing will fail (cannot serialize $fh).

   MCE::Shared->share(
      bless [ $path, $fh, \$order_id, \%hold ], $class
   );
}

sub send {
   ...
}

sub close {
   ...
}
[download]

Regarding Iter::Input, the construction makes a shared object by sending it to the shared-manager process, where it is managed.

package Iter::Input;

sub new {
   my ( $class, $chunk_size, $iterations ) = @_;
   my ( $chunk_id, $seq_a ) = ( 0, 1 );

   MCE::Shared->share(
      bless [ $iterations, $chunk_size, \$chunk_id, \$seq_a ], $class
   );
}

sub recv {
   ...
}
[download]

Replacing MCE::Hobo with threads is possible. One can run MCE::Shared alongside threads and threads::shared.

use threads;

...

test_threads();

sub test_threads {
   my $start = time;
   my @thrs;

   # must save threads to join later
   push @thrs, threads->create('work') for (1..$num_workers);

   # wait for threads to finish
   $_->join for @thrs;

   # close the shared file handle, flushes buffer
   $iter_o->close();

   printf STDERR "testa done in %0.02f seconds\n", time - $start;
}
[download]

Regards, Mario.