I'm analyzing our code base as an aid in refactoring. We incurred a lot of technical debt in the very early stages of our startup, and it's time we start paying it off.

The first step I took was to run everything through perltidy, so that I have consistent formatting throughout.

The second step was running Perl::Metrics::Simple on our code, which already reveals a lot of refactoring targets. The largest subs and the ones with the highest complexity will be first on my list.

The third step is the one I could use help with. The idea is to find duplicated code resulting from old-fashioned copy/paste-programming. What I think I need for this, is the inverse of a diff tool: something that finds blocks of code that are identical.

Do tools exist, that can find blocks of code occurring more than once in a batch of files?

I've begun with a naive indexing of every line, marking the file name and line number of every occurrence. This allows me to query the result in a variety of interesting ways. For example, here are the six most common lines: 
+----------------------------+-------+
| loc                        | refs  |
+----------------------------+-------+
|                            | 11381 |
| }                          |  4358 |
| {                          |  4121 |
| );                         |   746 |
| else                       |   475 |
| 1;                         |   404 |
+----------------------------+-------+

[download]
As you can see, we're using the BSD/Allman brace style.

If anyone is interested, I can show my data model, crossreferencing script, and the web-based browser I have made for this.

In reply to Refactoring tools for copy/paste jobs by rhesa

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.