Callback from threaded shared library

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I am working under Linux, using Perl 5.8.0, and I have a shared object that takes a callback, starts a pthread and when the thread finishes, calls the callback. This works fine if done in C.
However, I need to write a Perl extension (using XS and following mainly Strategies for storing callback) around this shared object, and I cannot get it to work properly.
I can register a callback to a Perl sub. I can call this sub all right as long as I'm in the main thread. However, if the thread calls the callback the whole thing crashes (segmentation fault), before it even enters the XS function that perpares and calls the Perl sub.
Here's some code that doesn't work (my excuses if it's too much code, I tried to keep it to a bare minumum):

My shared object mylib.c:

//gcc -o libmylib.so -shared -fpic mylib.c -lpthread
#include "mylib.h"
#include <pthread.h>

static lib_work_done_callback_t GCB;
static void *GData;

static void *run( void *Arg )
{
    sleep( 3 );
    printf( "Calling callback() from thread.\n" );
    GCB( GData ); /* crashes here */
    printf( "Calling callback() from thread done.\n" );
    return( NULL );
}

int lib_RegisterWorkDoneCallback( void *a, 
    lib_work_done_callback_t CB, void *Data )
{
    pthread_t thread;
    pthread_attr_t attr;
    int res = 0;
    
    GCB = CB;
    GData = Data;
    
    pthread_attr_init( &attr );
    pthread_attr_setdetachstate( &attr, PTHREAD_CREATE_DETACHED );
    if( pthread_create( &thread, &attr, run, NULL ) ) {
        printf( "Error: Could not create thread.\n" );
        res = 1;
    }
    pthread_attr_destroy( &attr );
    return( res );
}
[download]

The header for the library mylib.h:

#ifndef mylib_h
#define mylib_h
typedef void (*lib_work_done_callback_t)( void * );
int lib_RegisterWorkDoneCallback( void *a, 
    lib_work_done_callback_t CB, void *Data );
#endif
[download]

My XS MyLib.xs:

#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"

#include "ppport.h"

#include "../mylib.h"

#define MY_CXT_KEY "MyLib::_guts" XS_VERSION

typedef struct {
    HV * Mapping;
} my_cxt_t;

START_MY_CXT

static void
lib_RegisterWorkDoneCallback_if( void *Data )
{
  dSP;
  dMY_CXT;
  printf( "Performing perl callback.\n" ); fflush( stdout );
  ENTER;
  SAVETMPS;
  SV ** sv;
  sv = hv_fetch( MY_CXT.Mapping, (void*)&Data, sizeof(Data), FALSE );
  if( sv == (SV**)NULL)
    croak("Internal error. No callback registered.\n");
  PUSHMARK( sp );
  XPUSHs( sv_2mortal( newSViv( (int) Data ) ) );
  PUTBACK;
  perl_call_sv( *sv, G_DISCARD );
  FREETMPS;
  LEAVE;
}

#include "const-c.inc"

MODULE = MyLib        PACKAGE = MyLib                PREFIX = lib_

INCLUDE: const-xs.inc

BOOT:
{
    MY_CXT_INIT;
    MY_CXT.Mapping = (HV*)NULL;
}

int
lib_RegisterWorkDoneCallback(A, Callback, Data)
  void *A
  SV *Callback
  void *Data
  CODE:
  {
    dMY_CXT;
    printf( "Registering Perl callback.\n" ); fflush( stdout );
    if( MY_CXT.Mapping == (HV*)NULL )
      MY_CXT.Mapping = newHV();
    hv_store( MY_CXT.Mapping, (void*)&Data, sizeof( Data ),
                          newSVsv( Callback ), 0 );

    RETVAL = lib_RegisterWorkDoneCallback( A, 
            &lib_RegisterWorkDoneCallback_if, Data );
  }
  OUTPUT:
    RETVAL
[download]

My PM MyLib.pm:

package MyLib;
use 5.008;
use strict;
use warnings;
use Carp;
require Exporter;
use AutoLoader;
our @ISA = qw(Exporter);
our %EXPORT_TAGS = ( 'all' => [ qw(
) ] );
our @EXPORT_OK = ( @{ $EXPORT_TAGS{'all'} } );
our @EXPORT = qw(
);
our $VERSION = '0.01';
sub AUTOLOAD {
    my $constname;
    our $AUTOLOAD;
    ($constname = $AUTOLOAD) =~ s/.*:://;
    croak "&MyLib::constant not defined" if $constname eq 'constant';
    my ($error, $val) = constant($constname);
    if ($error) { croak $error; }
    {
    no strict 'refs';
        *$AUTOLOAD = sub { $val };
    }
    goto &$AUTOLOAD;
}
require XSLoader;
XSLoader::load('MyLib', $VERSION);
1;
__END__
[download]

Any tips or hints about what to do ? I saw a post (from 1999!) on google groups from Jan Dubois (Google groups) about something like this, but I have no idea how to wait for Perl's main thread, how to synchronise the two threads, if it at all is relevant today.
I've been hitting my head bloody over this; I hope someone can help.
Thanks!

Comment on Callback from threaded shared library Select or Download Code

Replies are listed 'Best First'.
Re: Callback from threaded shared library by PreferredUserName (Pilgrim) on Feb 23, 2005 at 15:44 UTC
You don't want to call into perl from two different threads at the same time. That will cause the crashing you're seeing. One approach would be to have an event queue that you pull off of from Perl, and push onto from the C callback. So instead of: `void my_callback(void data) { call_into_perl_event_handler(data); }` [download] do something like: // XS: void my_callback(void data) { pthread_mutex_lock(&event_queue_lock); push_onto_event_queue(data); pthread_mutex_unlock(&event_queue_lock); pthread_cond_broadcast(&event_queue_not_empty); } void get_next_event() PPCODE: pthread_mutex_lock(&event_queue_lock); while (event_queue_empty()) pthread_cond_wait(&event_queue_not_empty, &event_queue_loc +k); void *data = pull_off_event_queue(); pthread_mutex_unlock(&event_queue_lock); EXTEND(SP, 1); PUSHs(sv_2mortal(newSVpvn(data, ...))); # Perl: while (my $event = get_next_event()) { event_handler($event); } [download] Hope that helps.	[reply] [d/l] [select]
Re: Callback from threaded shared library by BrowserUk (Patriarch) on Feb 23, 2005 at 19:43 UTC
Basically, you cannot call perl code across threads. If you create an interpreter in the context of your main thread, call into it and retrieve a code reference, either directly, or indirectly in the form of method addresses attached to a returned object, attempting to call that/those address from a separate thread is going to crash. To allow multiple perl threads to run, a separate interpreter is started in each thread, and all the context they require to run is stored in thread-local storage. If you pass a callback generated in one interpreter (thread) across thread boundaries within your C code and then attempt to call it, the thread relative context will be missing and a crash is inevitable. Even if you started a second interpreter in the second thread, a callback created from within the first thread/interpreter will not work if passed to the second interpreter. The best option would be to start your interpreter in it's own thread and have the perl code in that thread go into an endless loop blocking on a read from a queue. Whenever your other (C) threads need something done by the perl thread, the post a token onto the queue that tells the perl code what is required. The C thread then blocks reading the queue until the perl code push the required results back onto the queue. You'll need to have the request tokens passed in carry an identifier (threadid) of the "calling thread" so that it can attach the same identifier to the results it posts for that thread. The C code reading the queue will need to discriminate between replies intended for it, and those intended for other calling threads, and leave any not intended for it on the queue and contniue blocking. This is the message loop model used by gui's like tk and others. Examine what is said, not who speaks. Silence betokens consent. Love the truth but pardon error.	[reply]