dgaramond2 has asked for the wisdom of the Perl Monks concerning the following question:

sub du { my @files; my $extra = 0; my $total = 0; my $recursive = 1; #my $blocksize = 1024; # XXX hardcoded #my $follow_symlinks = 0; for my $e (@_) { if (ref($e) eq 'HASH') { for (keys %$e) { if ($_ eq 'extra') { $extra = $e->{$_} } elsif ($_ eq 'total') { $total = $e->{$_} } #elsif ($_ eq 'blocksize') { $blocksize = $e->{$ +_} } #elsif ($_ eq 'follow_symlinks') { $follow_symlinks = +$e->{$_} } elsif ($_ eq 'recursive') { $recursive = $e->{$ +_} } } } else { push @files, $e; } } my @res; my %inodes; my $doit; $doit = sub { my ($files, $skip_dot_ddot, $recursive, $idx) = @_; my $i = 0; #print "\@", `pwd`, ": doit([".join(", ", @$files)."], skip_do +t_ddot=$skip_dot_ddot, recursive=$recursive, idx=$idx)\n"; for my $f (@$files) { #print "file #$i=$f\n"; #system "pwd"; next if $skip_dot_ddot && $f =~ /^\.\.?$/; my @st = stat($f) or next; my $is_dir = (-d _) && !(-l $f); my $j = $idx == -1 ? $i : $idx; my $counted = $inodes{"$st[0]:$st[1]"}++; if (!$res[$j]) { $res[$j] = [ 0, 0, 0, 0, 0 ]; # SIZE, FILES, DIRS, UNI +QUE_FILES, UNIQUE_DIRS } my $r = $res[$j]; $r->[0] += $st[7] unless $counted; if ($is_dir) { $r->[2]++; $r->[4]++ unless $counted; if ($recursive) { if (chdir $f) { opendir my $dh, "."; $doit->([readdir $dh], 1, 1, $j); chdir ".."; } } } else { $r->[1]++; $r->[3]++ unless $counted; } $i++; } }; $doit->(\@files, 0, $recursive, ($total ? 0 : -1)); if ($extra) { return map { {size=>$_->[0], files=>$_->[1], dirs=>$_->[2], un +ique_files=>$_->[3], unique_dirs=>$_->[4] } } @res; } else { return map { $_->[0] } @res; } } use Benchmark qw(:all); timethis(1, sub {@res=du(".")});

When pitted against the C-based "du" command on a tree with +- 150k entries:

Perl: 1.58s user, 0.46s sys C: 0.10s user, 0.47s sys

This means the Perl version is about 15 times less efficient. Any ideas on how to make it more efficient? I'll settle with 2-4x slower, but 15x is rather unsatisfactory for me.

Replies are listed 'Best First'.
Re: How to make this perl version of "du" faster?
by moritz (Cardinal) on Oct 28, 2009 at 08:40 UTC

    Install a decent profiler and run it on your code. Identify the hotspots. Optimize them.

    See also: perlperf.

    Perl 6 - links to (nearly) everything that is Perl 6.
Re: How to make this perl version of "du" faster?
by almut (Canon) on Oct 28, 2009 at 08:42 UTC

    I'd say it's not too unusual for a Perl program to be 15 times slower than an optimized C program with similar functionality.  I don't think there's anything glaringly obvious that would make your code slow, though you could probably find the one or the other possibility for minor optimisations...  For this, it's easiest to first profile the code to see where it spends most of its time — e.g. using Devel::NYTProf.

Re: How to make this perl version of "du" faster?
by Anonymous Monk on Oct 28, 2009 at 08:14 UTC
    don't shell out, use Cwd, or qx!du ...!
Re: How to make this perl version of "du" faster?
by JavaFan (Canon) on Oct 28, 2009 at 09:16 UTC
    Any ideas on how to make it more efficient?
    Write it in C, glue it with XS to Perl. You could probably lift the majority of the needed C code from an open source implementation of 'du'.
Re: How to make this perl version of "du" faster?
by Anonymous Monk on Oct 28, 2009 at 09:14 UTC
Re: How to make this perl version of "du" faster?
by toolic (Bishop) on Oct 28, 2009 at 13:18 UTC

      I did, and there's also File::Size, but both only total file sizes or blocks, disregarding hard links/duplicate inodes. And both do not have the equivalent of "du"'s -c option which I need.

      Anyway, the reason I wrote this is to count the number of files as well as the disk usage in one go, because I thought it might be more efficient than a call to "du" followed by "find".

      Turns out "du" has -a which can be used to count number of files/subdirs, so my current du() calls out the "du" command now. I'm moving my original du() to du_perl(), for later playing.

Re: How to make this perl version of "du" faster?
by gmargo (Hermit) on Oct 28, 2009 at 17:27 UTC

    I've done several tests with your code and am getting a pretty consistent 3.5 to 1 ratio. I think you may be failing to take filesystem caching into account. I ran your code and du multiple times on really large directory trees (~500k entries) and came up with 3.5x with their best times, so I think you have nothing to worry about.

    Update in response to dgaramond2's reply: I'm running a much wimpier system, Sempron 2700, Ubuntu 8.04 LTS, 2.6.24, ext3(ide).

      Don't think so, I've run the benchmark multiple times in succession, as well as reverse the order. Same thing. I'm using A64 5100+, Ubuntu 9.10 & Debian 5.0, Linux 2.6.31 & 2.6.26, ext3.

      Will try some more and also tried the obvious profiling route. Thanks for the reminder & all the responses.