in reply to Re: Help Diagnosing Memory Leak
in thread Help Diagnosing Memory Leak
I had already overwritten the offending code with a procedural version which is why I didn't provide it. I have however re-produced the problem below with a much less complex compare routine. I generated a file of 1 .. 100_000_000. I then split that into ten 10_000_000 line files. Then I ran perl merge.pl > source.merged and watched the memory usage continue to climb.
#!/usr/bin/perl use strict; use warnings; use List::Util 'first'; my $finish = 'The end is near'; my @list; for (glob('source_part_??')) { open(my $fh, '<', $_) or die "Unable to open '$_' for reading: $!" +; push @list, $fh; } my $fetch = sub { my ($fh) = @_; return $finish if eof $fh; return scalar <$fh>; }; my $compare = sub {$_[0] <=> $_[1]}; my $next = gen_merger(\@list, $fetch, $compare, $finish); while (1) { my $item = $next->(); last if defined $item && $item eq $finish; print "$item\n"; } sub gen_merger { my ($list, $fetch, $compare, $finish) = @_; my @item = map $fetch->($_), @$list; my $done; return sub { return $finish if $done; my $idx = first {$item[$_] ne $finish} 0 .. $#item; my $next = $item[$idx]; for ($idx + 1 .. $#item) { next if $item[$_] eq $finish; my $result = $compare->($next, $item[$_]); ($idx, $next) = ($_, $item[$_]) if $result == 1; } $item[$idx] = $fetch->($list->[$idx]); $done = 1 if ! defined first {$item[$_] ne $finish} 0 .. $#ite +m; return $next; }; }
Note: This version has one more bug fix in the line immediately preceding return $next. The range was changed from $idx .. $#item to 0 .. $#item.
Cheers - L~R
|
|---|
| Replies are listed 'Best First'. | |
|---|---|
|
Re^3: Help Diagnosing Memory Leak
by BrowserUk (Patriarch) on Mar 02, 2011 at 16:09 UTC | |
by Limbic~Region (Chancellor) on Mar 02, 2011 at 17:41 UTC |