I do not believe this is because of anything explicit in the C routines since there is only one malloc call, ... and such a leak would grow in a regular linear fashion every time a new memory page was needed.

That is a bad assumption. Perl's memory allocator often requests a power-of-two sized large chunk of memory from the OS which it then parcels out to the program as a bunch of smaller requests on demand.

My guess is that you are failing to mortalise the SV which you are populating with the inbound data.

If you posted the code, it would probably be the work of a few minutes to confirm that or otherwise track the leak down.

My guess is that you are failing to mortalise the SV which you are populating with the inbound data. If you posted the code, it would probably be the work of a few minutes to confirm that or otherwise track the leak down.

The code is too substantial to post, but maybe there is something I have wrong WRT to mortality.

#!/usr/bin/perl
use strict;
use warnings FATAL => qw(all);

use Inline 'C';

my $test = eg();

reportHash($test);
reportScalar($test->{x});

$test = undef;

__DATA__
__C__
SV *eg () {
    HV *rv = newHV();
    hv_store(rv, "x", 1, newSVpv("hello", 0), 0);
    return newRV_noinc((SV*)rv);
}

void reportHash (SV *ref) {
    HV *hash = (HV*)SvRV(ref);
    fprintf(stderr,"rc: %d\n", SvREFCNT((SV*)hash));
}

void reportScalar (SV *s) {
    fprintf(stderr,"rc: %d\n", SvREFCNT(s));
}
[download]

My understanding is that that since $test has a reference count of 1, and the SV inside it has a reference count of 1, that $test=undef should free the memory used. Running that in a while(1) loop seems to confirm this -- there is no growth. Where in this code do you see an issue? Everything else follows the same pattern (I am not using the inline stack).

That example code leaks an SV. You return a newly-minted RV that points to a HV. In the $test assignment, the value of that RV is copied to $test; the original RV leaks.

In your production code, try examining the value of the perl internals variable PL_sv_count (e.g. write a little C wrapper function that returns its value to perl space). This var shows the total number of SVs allocated. If it keeps growing, it means that you're leaking SVs.

Dave.

You might want to look into Devel::Peek:

#!/usr/bin/perl
use strict;
use warnings FATAL => qw(all);
use Inline 'C';
use Devel::Peek;

my $test = eg();

Dump( $test );

$test = undef;

__DATA__
__C__
SV *eg () {
    HV *rv = newHV();
    hv_store(rv, "x", 1, newSVpv("hello", 0), 0);
    return newRV_noinc((SV*)rv);
}
[download]

Outputs:

C:\test>junkic
SV = RV(0x267460) at 0x267450
  REFCNT = 1
  FLAGS = (PADMY,ROK)
  RV = 0x2d0200
  SV = PVHV(0x357b620) at 0x2d0200
    REFCNT = 1
    FLAGS = (SHAREKEYS)
    ARRAY = 0x36132d8  (0:7, 1:1)
    hash quality = 100.0%
    KEYS = 1
    FILL = 1
    MAX = 7
    RITER = -1
    EITER = 0x0
    Elt "x" HASH = 0x9303a5e5
    SV = PV(0x37cf50) at 0x37f128
      REFCNT = 1
      FLAGS = (POK,pPOK)
      PV = 0x36d2f18 "hello"\0
      CUR = 5
      LEN = 8
[download]

Which looks fine, but only serves to tell us that what you've posted isn't indicative of your actual code.

---

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

Are you storing anything between connections?

Perhaps you could print the length of the dumper string for any globals to the console, and see if they grow over time.

Are you storing anything between connections?

No. Connections and requests are hashes (in hashes), but these are deleted after the connection is closed or the request is completed. So, eg, if I send 100000 requests with a concurrency of 50, the number of buckets in the connections hash won't exceed 64, and the number of used buckets afterward is what I'd expect.

void Reference_recursivePurge (SV *ref, char *txt) {
    char label[1024];
    int rc;

    if (!SvROK(ref)) {
        rc = SvREFCNT(ref);
        if (rc > 1) {
            fprintf(stderr,"REFCOUNT for %s: %d\n", txt, rc);
            while (rc > 1) {
                SvREFCNT_dec(ref);
                rc--;
            }
        }
        return;
    }

    if (SvTYPE(SvRV(ref)) == SVt_PVAV) {
        AV *ray = (AV*)SvRV(ref);
        SV **cur;
        int i, len = av_len(ray);
        for (i = 0; i <= len; i++) {
            cur = av_fetch(ray, i, 1);
            sprintf(label, "%s[%d]", txt, i);
            Reference_recursivePurge(*cur, label);
            i++;
        }
        rc = SvREFCNT(ray);
        if (rc > 1) {
            fprintf(stderr,"REFCOUNT for %s: %d\n", txt, rc);
            while (rc > 1) {
                SvREFCNT_dec((SV*)ray);
                rc--;
            }
        }
        return;
    }

    if (SvTYPE(SvRV(ref)) == SVt_PVHV) {
        HV *hash = (HV*)SvRV(ref);
        SV *cur;
        char *key;
        int len;
        hv_iterinit(hash);
        while ((cur = hv_iternextsv(hash, &key, &len))) {
            sprintf(label, "%s->%s", txt, key);
            Reference_recursivePurge(cur, label);
        }
        rc = SvREFCNT(hash);
        if (rc > 1) {
            fprintf(stderr,"REFCOUNT for %s: %d\n", txt, rc);
            while (rc > 1) {
                SvREFCNT_dec((SV*)hash);
                rc--;
            }
        }
    }
}
[download]