I've been debugging some code which uses a huge amount of memory on a machine where memory is short and the processes involved cause the system to swap and performance to degrade awefully. After much investigation the problem code appears to be:
eval {
$db->execProc(
$p_job_finished,
{DieOnError => 1},
$job->{jobid}, $job->{jobsts},
(exists($job->{jobnative}) ? $job->{jobnative} : undef),
(exists($job->{joberr}) ?
substr($job->{joberr}, 0, DB_MAX_CHRS_NOT_LONG) : und
+ef),
(defined($job->{jobinterr}) ?
substr($job->{jobinterr}, 0, DB_MAX_CHRS_NOT_LONG) :
+undef),
$repost, $clientref,
substr(Dumper($results), 0, DB_MAX_CHRS_NOT_LONG));
1;
};
where $db is a wrapper around DBI adding an execProc method which calls a procedure which inserts some data into a table and DB_MAX_CHRS_NOT_LONG is 4000. When this code was first written the $results array ref would not be very large and a Dumper of it was always shorter then 4000 and the substr was simply a protection against an attempt to insert too much into the column. However, changes to the system now mean the $results array ref can be very large (sometimes as big as 16Mb when dumped using Data::Dumper). When the Dumper above is run the resident set size rises to around 420Mb from 100Mb and this is causing the problem and worse only the first 4K of the dump is used anyway.
Only a sample of the $results array ref is required (at most 4K worth) and the resulting dump is only for debugging/auditing and does not need to be converted back to a Perl structure. The obvious thing to do it to stop creating the dump string at 4K but Data::Dumper does not seem to support that. I tried Data::Dump::Partial but it is awefully slow (less than 1 per second compared with Dumper at 60 a second on the same, smallish array reference). $results only contains arrays of around 3 or 4 levels deep and simple string/number scalars? Other than roll my own (which is a possibility) is there anything else that produces a readable string version of a partial Perl structure?
