in reply to Tracing memory leak

I do not believe this is because of anything explicit in the C routines since there is only one malloc call, ... and such a leak would grow in a regular linear fashion every time a new memory page was needed.

That is a bad assumption. Perl's memory allocator often requests a power-of-two sized large chunk of memory from the OS which it then parcels out to the program as a bunch of smaller requests on demand.

My guess is that you are failing to mortalise the SV which you are populating with the inbound data.

If you posted the code, it would probably be the work of a few minutes to confirm that or otherwise track the leak down.


Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

Replies are listed 'Best First'.
Re^2: Tracing memory leak
by halfcountplus (Hermit) on Sep 14, 2011 at 19:18 UTC

    My guess is that you are failing to mortalise the SV which you are populating with the inbound data. If you posted the code, it would probably be the work of a few minutes to confirm that or otherwise track the leak down.

    The code is too substantial to post, but maybe there is something I have wrong WRT to mortality.

    #!/usr/bin/perl use strict; use warnings FATAL => qw(all); use Inline 'C'; my $test = eg(); reportHash($test); reportScalar($test->{x}); $test = undef; __DATA__ __C__ SV *eg () { HV *rv = newHV(); hv_store(rv, "x", 1, newSVpv("hello", 0), 0); return newRV_noinc((SV*)rv); } void reportHash (SV *ref) { HV *hash = (HV*)SvRV(ref); fprintf(stderr,"rc: %d\n", SvREFCNT((SV*)hash)); } void reportScalar (SV *s) { fprintf(stderr,"rc: %d\n", SvREFCNT(s)); }
    My understanding is that that since $test has a reference count of 1, and the SV inside it has a reference count of 1, that $test=undef should free the memory used. Running that in a while(1) loop seems to confirm this -- there is no growth. Where in this code do you see an issue? Everything else follows the same pattern (I am not using the inline stack).
      That example code leaks an SV. You return a newly-minted RV that points to a HV. In the $test assignment, the value of that RV is copied to $test; the original RV leaks.

      In your production code, try examining the value of the perl internals variable PL_sv_count (e.g. write a little C wrapper function that returns its value to perl space). This var shows the total number of SVs allocated. If it keeps growing, it means that you're leaking SVs.

      Dave.

        That example code leaks an SV.

        No, it does not. As mentioned, running this within a while(1) loop, even for several minutes (ie, millions and millions of iterations) produces 0 growth. That is unequivocally no leak.

        my $count = 0; while (1) { my $test = eg(); print $count++."\n"; reportHash($test); reportScalar($test->{x}); $test = undef; }

        Try this yourself. If even a single byte were leaking per iteration, this would be pretty obvious pretty quickly in a monitor.

        I believe the problem you perceive is circumvented by the use of _noinc -- the value returned does not increment the reference count of the SV. Otherwise, it would be impossible to return an RV without a leak.

        I will look at PL_sv_count tho.

      You might want to look into Devel::Peek:

      #!/usr/bin/perl use strict; use warnings FATAL => qw(all); use Inline 'C'; use Devel::Peek; my $test = eg(); Dump( $test ); $test = undef; __DATA__ __C__ SV *eg () { HV *rv = newHV(); hv_store(rv, "x", 1, newSVpv("hello", 0), 0); return newRV_noinc((SV*)rv); }

      Outputs:

      C:\test>junkic SV = RV(0x267460) at 0x267450 REFCNT = 1 FLAGS = (PADMY,ROK) RV = 0x2d0200 SV = PVHV(0x357b620) at 0x2d0200 REFCNT = 1 FLAGS = (SHAREKEYS) ARRAY = 0x36132d8 (0:7, 1:1) hash quality = 100.0% KEYS = 1 FILL = 1 MAX = 7 RITER = -1 EITER = 0x0 Elt "x" HASH = 0x9303a5e5 SV = PV(0x37cf50) at 0x37f128 REFCNT = 1 FLAGS = (POK,pPOK) PV = 0x36d2f18 "hello"\0 CUR = 5 LEN = 8

      Which looks fine, but only serves to tell us that what you've posted isn't indicative of your actual code.


      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.

        Which looks fine, but only serves to tell us that what you've posted isn't indicative of your actual code.

        AFAICT it is, but you can judge for yourself. There are two structures that get repeatedly created and destroyed. A connection object (this is a slightly unorthodox constructor since it is a class method of a base common to the client and server):

        SV *acceptConnection (SV *objref) { char error[1024]; HV *self = (HV*)SvRV(objref), *client = newHV(); SV **field = hv_fetch(self, "fd", 2, 0); int srv = SvIV(*field), sock, flags, yes = 1; struct sockaddr_in info; socklen_t len = sizeof(struct sockaddr_in); sock = accept(srv, (struct sockaddr*)&info, &len); if (sock == -1) { snprintf(error, 1024, "accept(): %s", strerror(errno)); hv_store(client, "sockerr", 7, newSVpv(error, 0), 0); close(sock); } else { setsockopt(sock, SOL_SOCKET, SO_KEEPALIVE, &yes, sizeof(int)); hv_store(client, "fd", 2, newSViv(sock), 0); hv_store(client, "ip", 2, newSVpv(inet_ntoa(info.sin_addr), 0) +, 0); flags = fcntl(sock, F_GETFL, 0); if (flags == -1) { snprintf(error, 1024, "fcntl(F_GETFL): %s", strerror(errno +)); hv_store(client, "sockerr", 7, newSVpv(error, 0), 0); close(sock); } else if (fcntl(sock, F_SETFL, flags | O_NONBLOCK) == -1) { snprintf(error, 1024, "fcntl(F_SETFL O_NONBLOCK): %s", str +error(errno)); hv_store(client, "sockerr", 7, newSVpv(error, 0), 0); close(sock); } else { hv_store(client, "events", 6, newSViv(SPFD_READ), 0); hv_store(client, "state", 5, newSViv(READ_HEAD), 0); hv_store(client, "readerr", 7, newSViv(0), 0); hv_store(client, "errcount", 8, newSViv(0), 0); hv_store(client, "queue", 5, newRV_noinc((SV*)newAV()), 0) +; } } return sv_bless(newRV_noinc((SV*)client), gv_stashpv("Sloop::Socke +t", 0)); }

        And a request hash, produced by "parseRequest()" which contains a hash produced by "parseHeaders()":

        HV *parseHeaders (char **data) { HV *hdrs = newHV(); char copy[8192], *p, *val = NULL, *str = *data; int state = NAME; SV **cur; while (1) { // extract field name if (state == NAME) { if (sscanf(str, "%[^:]:", copy) != 1) break; // malform +ed lowerCase(copy); cur = hv_fetch(hdrs, copy, strlen(copy), 1); if (!SvOK(*cur)) *cur = newSVpv("", 0); state = VALUE; continue; } // extract field value if (!val) { p = val = strchr(str, ':'); val++; while (*val == ' ' || *val == '\t') val++; } if (!(*p)) break; // malformed if (p[0] == '\r' && p[1] == '\n') { p += 2; if (*p > 32 || (p[0] == '\r' && p[1] == '\n')) { // next line starts with a new name or \r\n, so field is c +omplete *(p-2) = '\0'; if (SvCUR(*cur)) { // multiple entries val--; *val = ','; } sv_catpv(*cur, val); if (p[0] == '\r' && p[1] == '\n') break; // good val = NULL; state = NAME; str = p; } } else p++; } *data = p; return hdrs; } HV *parseRequest (char *raw, int stop) { int len, state = METHOD; char *p = raw, *val, *qs; SV **field; HV *rv = newHV(); hv_store(rv, "err", 3, newSViv(INCOMPLETE), 0); while (p - raw < stop) { // HTTP 1.x Request Line extracted in METHOD->PATH->PROTOCOL state +s if (state == METHOD) { while (*p == ' ' || *p == '\t') p++; val = p; while (*p != ' ' && *p != '\t' && *p != '\0') p++; *p = '\0'; lowerCase(val); hv_store(rv, "method", 6, newSVpv(val, 0), 0); p++; state = PATH; continue; // loop condition checks bounds ;) } if (state == PATH) { while (*p <= 32 && *p > 0) p++; if (*p != '/' && *p != '*') { // must be an absolute path or * (RFC 2616) field = hv_fetch(rv, "err", 3, 0); sv_setiv(*field, BAD_PATH); break; } // turn path into an array AV *path = newAV(); val = p+1; if (*val <= 32) { av_push(path, newSVpv(p, 1)); p++; } else { while (*p > 32) p++; *p = '\0'; // check for query string if ((qs = strrchr(val, '?'))) { *qs = '\0'; qs = queryString(rv, ++qs); } while ((p = strchr(val, '/'))) { if (!*(p+1)) break;// path ends with '/' av_push(path, newSVpv(val, p-val)); val = ++p; } len = strlen(val); if (qs) p = qs + 1; else p = val + len + 1; if (val[len-1] == '/') len--; av_push(path, newSVpv(val, len)); } hv_store(rv, "path", 4, newRV_noinc((SV*)path), 0); state = PROTOCOL; continue; } if (state == PROTOCOL) { while (*p == ' ' || *p == '\t') p++; if (*p < 32) // no protocol hv_store(rv, "protocol", 8, newSVpv("???", 0), 0); else { val = p; while (*p > 32) p++; *(p++) = '\0'; hv_store(rv, "protocol", 8, newSVpv(val, 0), 0); } while (*p <= 32 && *p) p++; break; } } if (state >= PROTOCOL) { hv_store(rv, "headers", 7, newRV_noinc((SV*)parseHeaders(&p)), + 0); while (*p == '\r' || *p == '\n') p++; if (*p && p < raw + stop) hv_store(rv, "body", 4, newSVpv(p, s +top - (p-raw)), 0); field = hv_fetch(rv, "err", 3, 0); sv_setiv(*field, OK); } return rv; }