Jarkko Hietaniemi used http://pmd.sourceforge.net/cpd.html to do duplicate code detection against the Perl source code, but I'm not sure that this tool will work well with Perl as it does a lot of parsing. Google found me another paper on lightweight detection of duplicated code, which sounds easy to implement and use with Perl. The trick in the paper seems to be to use a sliding window over sections of code and then to do a bag comparison, to protect against swapped lines etc.

Thanks, Corion. That paper certainly gives me a good starting point. For some reason, I could not come up with useful keywords to search for on http://search.cpan.org.

Phil

I'd actually start from the other end, looking for infrequently-occuring-but-nonunique lines. A line which is used in exactly 2 or 3 places is more likely to be part of a block of cut and paste than one used in 100.

When you are refactoring, a nice option you have in perl is to make use of first class functions/closures/anonymous subs.

This allows you to have more "fuzzy matching" between blocks of similar-but-not-identical code. You can pull out the shared boilerplate as a new method/sub and pass into it a closure which does the 'bit which is different for each occurence'.

It's a lightweight alternative to putting together an inheritance hierarchy to share code, which you may see in design patterns etc. For example:

sub print_tree
{
  my $tree = shift;
  print_tree($tree->left);
  print $tree->value;
  print_tree($tree->right);
}

sub sum_tree
{
  my $tree = shift;
  return sum_tree($tree->left)
         + $tree->node
         + sum_tree($tree->right);
}
[download]

sub walk_tree
{
  my $tree = shift;
  my $per_node = shift;
  walk_tree($tree->left);
  $per_node->($tree->node);
  walk_tree($tree->right);
}

sub print_tree
{
  my $tree = shift;
  walk_tree($tree, sub { print $_[0]; });
}

sub sum_tree
{
  my $tree = shift;
  my $total = 0;
  walk_tree($tree, sub { $total += $_[0]; });
  return $total;
}
[download]

As usual, it's entirely possible I'm way off base here... But I'm thinking of using something like Ted Pedersen's Ngram Statistics Package to implement my own version of Amazon's "Statistically Improbable Phrases," a technique that might work well for the OP in regard to his question.

You may also want to Super Search for "ngrams", "concordance" or "KWIC" (key word in context). My favorite KWIC is collocate, but there are several versions here on PM as well.

Also, I don't want to let you go without my recommending Doxygen.

Doxygen generates JavaDoc-like documentation for projects in many languages, including C and C++. DoxygenFilter is a new take on an old project (DoxyFilt) that also handles ("filters") Perl code, and now promises support for multi-programming-language projects. See Examples of output generated by doxygen.

HTH,

planetscape

I'd actually start from the other end, looking for infrequently-occuring-but-nonunique lines. A line which is used in exactly 2 or 3 places is more likely to be part of a block of cut and paste than one used in 100.

That's a great suggestion, I hadn't thought of that yet. Thanks!

I'm well aware of the various refactoring techniques, and I do use both OO and functional approaches where appropriate. Your example is illustrative enough though, although your walk_tree should probably read:

sub walk_tree
{
  my $tree = shift;
  my $per_node = shift;
  walk_tree($tree->left);
  $per_node->($tree->node);
  walk_tree($tree->right);
}
[download]

Thanks for the correction (which I've incorporated into the original node - this reply is partly to record that fact).

And I'm sorry if I seemed to imply that you might not be aware of such techniques - no offence intended.