I've been debugging some code which uses a huge amount of memory on a machine where memory is short and the processes involved cause the system to swap and performance to degrade awefully. After much investigation the problem code appears to be:

    eval {
        $db->execProc(
            $p_job_finished,
            {DieOnError => 1},
            $job->{jobid}, $job->{jobsts},
            (exists($job->{jobnative}) ? $job->{jobnative} : undef),
            (exists($job->{joberr}) ?
                 substr($job->{joberr}, 0, DB_MAX_CHRS_NOT_LONG) : und
+ef),
            (defined($job->{jobinterr}) ?
                 substr($job->{jobinterr}, 0, DB_MAX_CHRS_NOT_LONG) : 
+undef),
            $repost, $clientref,
        substr(Dumper($results), 0, DB_MAX_CHRS_NOT_LONG));
        1;
    };

[download]

where $db is a wrapper around DBI adding an execProc method which calls a procedure which inserts some data into a table and DB_MAX_CHRS_NOT_LONG is 4000. When this code was first written the $results array ref would not be very large and a Dumper of it was always shorter then 4000 and the substr was simply a protection against an attempt to insert too much into the column. However, changes to the system now mean the $results array ref can be very large (sometimes as big as 16Mb when dumped using Data::Dumper). When the Dumper above is run the resident set size rises to around 420Mb from 100Mb and this is causing the problem and worse only the first 4K of the dump is used anyway.

Only a sample of the $results array ref is required (at most 4K worth) and the resulting dump is only for debugging/auditing and does not need to be converted back to a Perl structure. The obvious thing to do it to stop creating the dump string at 4K but Data::Dumper does not seem to support that. I tried Data::Dump::Partial but it is awefully slow (less than 1 per second compared with Dumper at 60 a second on the same, smallish array reference). $results only contains arrays of around 3 or 4 levels deep and simple string/number scalars? Other than roll my own (which is a possibility) is there anything else that produces a readable string version of a partial Perl structure?

In reply to Replacement for substr(Data::Dumper($x), 0, 4000) by mje

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.