I don't see a principal problem with the Perl stack and your function getting called again before it has completed, so I suppose the problem lies elsewhere (it's hard to tell though when only seeing a snippet of code taken out of its context).

One potential problem could be that you aren't caring about the return value of call_sv() (which is the number of items returned by the Perl routine). This is dangerous, because if the routine - for whatever reason - leaves a different number of items on the stack than you're expecting/handling (you're handling exactly one item with your POPs), the stack will be messed up...   (not necessarily your problem here, but still something to be aware of).

Yes, in fact, the complete code contains this:

int count;
...
count = call_sv(..., G_SCALAR);
assert(count==1);
[download]

which, AFAIK, should be "safe".

When an hashref is really a blessed object, I want its toString() method to be called to dump the hashref content to my result SV. Calling the object's toString() method is the task for the code I shown you.

But why do you need to go to XS and back to do that? I appreciate this is probably a simplified example but if the thing you want to serialise already has a toString method to serialise it in Perl AND you want to return this in your results SV why do that in a round about way in XS?

sub serialise {
    my $ref = shift;
    my $str = undef;
    eval{ $str = $ref->toString };
    return $str if defined $str;
    $str = XS_serialise($ref);
    return $str;
}
[download]

Yes, indeed. This should be the approach. However, (there's always one :), I want the serialization function to handle a big data structure, which can contain these blessed objects rather deep inside it.

Since the main purpose of all this mess is to speed up things, I thought I'd want to pass a reference to this "big structure" and then the XS function would do the heavy work.

Doing as you say, and I've tried that before, probably means that I must move all the data structure traversal outside of XS. Example:

my $big_struct = [
   {a=>1}, {b=>2},
   [1, 2, 3, 4, 5],
   ['blah', 'blah', ['bluh', $blessed_object]],
   {d=>3, e=>[1,2,3]}
];
[download]

If I want to correctly handle $blessed_object from XS, I must call its toString() (assume for now it has one) with call_sv().
Doing this from Perl code, means I must shift all the structure traversal from XS to Perl... losing a share of the speed gain...

Here is an example. It's done in Inline but is just straight vanilla XS. The stack pointer behaves itself over the separate calls.

package Serialize;
use Data::Dump;

sub new { bless {foo=>'bar'}, shift }
sub toString { return Data::Dump::dump($_[0]) } 
my $obj = new Serialize;
my $str = "some string";
print serialize($obj), $/;
print serialize($str), $/;

use Inline Config => FORCE_BUILD => 1, CLEAN_AFTER_BUILD => 0;
use Inline C => <<'END_OF_C_CODE';

void serialize(SV* sv_data) {
    int count;
    dXSARGS;
    sp = mark;
    printf("\nEntry SP is %d\n", sp);
    if (sv_isobject(sv_data)) {
        // this is an object so presume a toString method
        PUSHMARK(sp);
        XPUSHs(sv_data);  // whack it back on the stack
        PUTBACK;
        count = perl_call_pv("toString", G_SCALAR);
        if ( count != 1 ) 
            croak("Expected 1 value got %d\n", count);
        XPUSHs(sv_2mortal(newSVsv(POPs)));
    }
    else {
        // just return SV with something appended for this example
        sv_catpv(sv_data, " - serialized\0");
        XPUSHs(sv_2mortal(newSVsv(sv_data)));  
    }
    PUTBACK;
    printf("Exit  SP is %d\n", sp);
    XSRETURN(1);
}
END_OF_C_CODE

__DATA__
bless({ foo => "bar" }, "Serialize")
some string - serialized

Entry SP is 32203548
Exit  SP is 32203552

Entry SP is 32203548
Exit  SP is 32203552
[download]