in reply to Re: Tracing memory leak
in thread Tracing memory leak

My guess is that you are failing to mortalise the SV which you are populating with the inbound data. If you posted the code, it would probably be the work of a few minutes to confirm that or otherwise track the leak down.

The code is too substantial to post, but maybe there is something I have wrong WRT to mortality.

#!/usr/bin/perl use strict; use warnings FATAL => qw(all); use Inline 'C'; my $test = eg(); reportHash($test); reportScalar($test->{x}); $test = undef; __DATA__ __C__ SV *eg () { HV *rv = newHV(); hv_store(rv, "x", 1, newSVpv("hello", 0), 0); return newRV_noinc((SV*)rv); } void reportHash (SV *ref) { HV *hash = (HV*)SvRV(ref); fprintf(stderr,"rc: %d\n", SvREFCNT((SV*)hash)); } void reportScalar (SV *s) { fprintf(stderr,"rc: %d\n", SvREFCNT(s)); }
My understanding is that that since $test has a reference count of 1, and the SV inside it has a reference count of 1, that $test=undef should free the memory used. Running that in a while(1) loop seems to confirm this -- there is no growth. Where in this code do you see an issue? Everything else follows the same pattern (I am not using the inline stack).

Replies are listed 'Best First'.
Re^3: Tracing memory leak
by dave_the_m (Monsignor) on Sep 14, 2011 at 20:04 UTC
    That example code leaks an SV. You return a newly-minted RV that points to a HV. In the $test assignment, the value of that RV is copied to $test; the original RV leaks.

    In your production code, try examining the value of the perl internals variable PL_sv_count (e.g. write a little C wrapper function that returns its value to perl space). This var shows the total number of SVs allocated. If it keeps growing, it means that you're leaking SVs.

    Dave.

      That example code leaks an SV.

      No, it does not. As mentioned, running this within a while(1) loop, even for several minutes (ie, millions and millions of iterations) produces 0 growth. That is unequivocally no leak.

      my $count = 0; while (1) { my $test = eg(); print $count++."\n"; reportHash($test); reportScalar($test->{x}); $test = undef; }

      Try this yourself. If even a single byte were leaking per iteration, this would be pretty obvious pretty quickly in a monitor.

      I believe the problem you perceive is circumvented by the use of _noinc -- the value returned does not increment the reference count of the SV. Otherwise, it would be impossible to return an RV without a leak.

      I will look at PL_sv_count tho.

        No, it does not
        Ah, you're right. XS does an implicit sv_2mortal() of the returned RV, which is why it doesn't get leaked. I guess Inline::C must do something similar (I've never used Inline::C). The problem I perceived however wouldn't have been circumvented by the _noinc usage.

        Dave.

Re^3: Tracing memory leak
by BrowserUk (Patriarch) on Sep 14, 2011 at 19:43 UTC

    You might want to look into Devel::Peek:

    #!/usr/bin/perl use strict; use warnings FATAL => qw(all); use Inline 'C'; use Devel::Peek; my $test = eg(); Dump( $test ); $test = undef; __DATA__ __C__ SV *eg () { HV *rv = newHV(); hv_store(rv, "x", 1, newSVpv("hello", 0), 0); return newRV_noinc((SV*)rv); }

    Outputs:

    C:\test>junkic SV = RV(0x267460) at 0x267450 REFCNT = 1 FLAGS = (PADMY,ROK) RV = 0x2d0200 SV = PVHV(0x357b620) at 0x2d0200 REFCNT = 1 FLAGS = (SHAREKEYS) ARRAY = 0x36132d8 (0:7, 1:1) hash quality = 100.0% KEYS = 1 FILL = 1 MAX = 7 RITER = -1 EITER = 0x0 Elt "x" HASH = 0x9303a5e5 SV = PV(0x37cf50) at 0x37f128 REFCNT = 1 FLAGS = (POK,pPOK) PV = 0x36d2f18 "hello"\0 CUR = 5 LEN = 8

    Which looks fine, but only serves to tell us that what you've posted isn't indicative of your actual code.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

      Which looks fine, but only serves to tell us that what you've posted isn't indicative of your actual code.

      AFAICT it is, but you can judge for yourself. There are two structures that get repeatedly created and destroyed. A connection object (this is a slightly unorthodox constructor since it is a class method of a base common to the client and server):

      SV *acceptConnection (SV *objref) { char error[1024]; HV *self = (HV*)SvRV(objref), *client = newHV(); SV **field = hv_fetch(self, "fd", 2, 0); int srv = SvIV(*field), sock, flags, yes = 1; struct sockaddr_in info; socklen_t len = sizeof(struct sockaddr_in); sock = accept(srv, (struct sockaddr*)&info, &len); if (sock == -1) { snprintf(error, 1024, "accept(): %s", strerror(errno)); hv_store(client, "sockerr", 7, newSVpv(error, 0), 0); close(sock); } else { setsockopt(sock, SOL_SOCKET, SO_KEEPALIVE, &yes, sizeof(int)); hv_store(client, "fd", 2, newSViv(sock), 0); hv_store(client, "ip", 2, newSVpv(inet_ntoa(info.sin_addr), 0) +, 0); flags = fcntl(sock, F_GETFL, 0); if (flags == -1) { snprintf(error, 1024, "fcntl(F_GETFL): %s", strerror(errno +)); hv_store(client, "sockerr", 7, newSVpv(error, 0), 0); close(sock); } else if (fcntl(sock, F_SETFL, flags | O_NONBLOCK) == -1) { snprintf(error, 1024, "fcntl(F_SETFL O_NONBLOCK): %s", str +error(errno)); hv_store(client, "sockerr", 7, newSVpv(error, 0), 0); close(sock); } else { hv_store(client, "events", 6, newSViv(SPFD_READ), 0); hv_store(client, "state", 5, newSViv(READ_HEAD), 0); hv_store(client, "readerr", 7, newSViv(0), 0); hv_store(client, "errcount", 8, newSViv(0), 0); hv_store(client, "queue", 5, newRV_noinc((SV*)newAV()), 0) +; } } return sv_bless(newRV_noinc((SV*)client), gv_stashpv("Sloop::Socke +t", 0)); }

      And a request hash, produced by "parseRequest()" which contains a hash produced by "parseHeaders()":

      HV *parseHeaders (char **data) { HV *hdrs = newHV(); char copy[8192], *p, *val = NULL, *str = *data; int state = NAME; SV **cur; while (1) { // extract field name if (state == NAME) { if (sscanf(str, "%[^:]:", copy) != 1) break; // malform +ed lowerCase(copy); cur = hv_fetch(hdrs, copy, strlen(copy), 1); if (!SvOK(*cur)) *cur = newSVpv("", 0); state = VALUE; continue; } // extract field value if (!val) { p = val = strchr(str, ':'); val++; while (*val == ' ' || *val == '\t') val++; } if (!(*p)) break; // malformed if (p[0] == '\r' && p[1] == '\n') { p += 2; if (*p > 32 || (p[0] == '\r' && p[1] == '\n')) { // next line starts with a new name or \r\n, so field is c +omplete *(p-2) = '\0'; if (SvCUR(*cur)) { // multiple entries val--; *val = ','; } sv_catpv(*cur, val); if (p[0] == '\r' && p[1] == '\n') break; // good val = NULL; state = NAME; str = p; } } else p++; } *data = p; return hdrs; } HV *parseRequest (char *raw, int stop) { int len, state = METHOD; char *p = raw, *val, *qs; SV **field; HV *rv = newHV(); hv_store(rv, "err", 3, newSViv(INCOMPLETE), 0); while (p - raw < stop) { // HTTP 1.x Request Line extracted in METHOD->PATH->PROTOCOL state +s if (state == METHOD) { while (*p == ' ' || *p == '\t') p++; val = p; while (*p != ' ' && *p != '\t' && *p != '\0') p++; *p = '\0'; lowerCase(val); hv_store(rv, "method", 6, newSVpv(val, 0), 0); p++; state = PATH; continue; // loop condition checks bounds ;) } if (state == PATH) { while (*p <= 32 && *p > 0) p++; if (*p != '/' && *p != '*') { // must be an absolute path or * (RFC 2616) field = hv_fetch(rv, "err", 3, 0); sv_setiv(*field, BAD_PATH); break; } // turn path into an array AV *path = newAV(); val = p+1; if (*val <= 32) { av_push(path, newSVpv(p, 1)); p++; } else { while (*p > 32) p++; *p = '\0'; // check for query string if ((qs = strrchr(val, '?'))) { *qs = '\0'; qs = queryString(rv, ++qs); } while ((p = strchr(val, '/'))) { if (!*(p+1)) break;// path ends with '/' av_push(path, newSVpv(val, p-val)); val = ++p; } len = strlen(val); if (qs) p = qs + 1; else p = val + len + 1; if (val[len-1] == '/') len--; av_push(path, newSVpv(val, len)); } hv_store(rv, "path", 4, newRV_noinc((SV*)path), 0); state = PROTOCOL; continue; } if (state == PROTOCOL) { while (*p == ' ' || *p == '\t') p++; if (*p < 32) // no protocol hv_store(rv, "protocol", 8, newSVpv("???", 0), 0); else { val = p; while (*p > 32) p++; *(p++) = '\0'; hv_store(rv, "protocol", 8, newSVpv(val, 0), 0); } while (*p <= 32 && *p) p++; break; } } if (state >= PROTOCOL) { hv_store(rv, "headers", 7, newRV_noinc((SV*)parseHeaders(&p)), + 0); while (*p == '\r' || *p == '\n') p++; if (*p && p < raw + stop) hv_store(rv, "body", 4, newSVpv(p, s +top - (p-raw)), 0); field = hv_fetch(rv, "err", 3, 0); sv_setiv(*field, OK); } return rv; }

        There is too much missing from what you've posted to be able to mock something up to allow it to compile.

        The most productive technique I've found for tracking down such leaks is a simple, if rather laborious one of commenting out most of the body of the code -- for example, so that it returns just an empty hash or even just a scalar -- and checking that it no longer leaks. Then put it back in chunks until the leak reappears. Not sophisticated, but reliable.


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.