Just for clarity, there are only 2 threads in here. Main thread - calling MyRegister() and MyDeregister(). A callbackworker thread is spawned inside the library to fire callbacks. Random sleep is introduced in both threads to simulate scenario when both might wake up next to each other and run into race conditions somewhere inside the interpreter code.

Okay:

1. Your main thread sets up and runs a perl interpreter instance in the normal way.
2. From that interpreter you call into C passing a perl code reference taken within the auspices of that main thread.
3. Within the C code, you spawn another OS thread running pure C code and pass on the coderef and an interpreter context from the main thread.
4. The main thread then diddles around doing not very much, mostly sleeping and occasionally printing something to standard out.
5. The C thread mostly sleeps and occasionally calls back into perl using the context from the main thread.

Which means that every now and again both your main thread and your C thread are using the same perl interpreter context concurrently and errors occur.

Well yes. They would wouldn't they.

The whole purpose of iThreads (Interpreter threads) is to prevent that from happening, and you are circumventing that. Perl's internals were never designed for threading and are not reentrant because the interpreter relies upon a large block of static data for its operation. Just as many non-threaded C programs do.

When iThreading was added, the mechanism chosen to work around the lack of reentrancy within the interpreter, was to give each thread its own copy of that big block of static data (otherwise known as a "context"), thereby allowing separate interpreters to run in each thread without stomping all over each others data. By allowing a C thread to (re)use a context from an existing thread concurrently to that thread, you are bypassing that protection mechanism and things will obviously break.

Meanwhile, I tried Win32::Event to wait for callback in the main thread. The event is Set at the end of the callback subroutine. It worked fantastic.

It may appear so in your essentially do-nothing demo, but I can assure you, it isn't in reality. What (I assume) you are doing is effectively serialising access to the shared context so that whilst the callback is using it, the main thread is doing nothing but blocking in a wait state, and vice versa. And whilst that may appear to work given the limited operations of the demo, it will not work once you start trying to do something useful with it.

A simple example of how this will go wrong. One of the the many things stored in the context is the threads current working directory. Let's say your main thread was working its way through the files in the current directory when the callback fires on the C thread. Your semaphore stops it in its tracks and the callback runs. One of the things the callback does is change the current directory. Whilst the semaphore has prevented any immediate panics or other internal corruptions, when the callback ends and the main thread tries to pick up where it left off, it finds itself in a completely different directory to where it was, and everything goes tits up.

Or, your main thread is iterating a hash using each; the callback fires and adds or deletes keys from that hash. When the main thread runs again things go tits up.

Or, the main thread is incrementing an integer: ++$i, it has read the value of the integer into a register and incremented it when it is interrupted by the callback firing. When the callback finishes and it gets control again, it attempts to write the incremented value back to the variable, but the callback undef'd it. Blam!

The only way you will get away with calling back into the main thread from the C thread, is if the main thread does nothing at all (except sleep) once it has registered the callback. And that kind of defeats the purpose of having two threads.

iThreads work surprisingly well given their nature, but you mess with their internals at your own risk.