Hi

I'm meditating about a regex based heuristic to roughly detect if a text paragraph (multilines delimited by '\n\n') is rather perl source code than normal text.

The best idea I had so far was: using regexes to count the line endings with ';' or '}' possibly followed with a '#' part.

Another to check the frequency of words starting with a sigil.

I'm not talking about a valid parser, just a fuzzy detector.

Any better ideas?

One use case could be a JS that checks the contents of a posting in the monastery and warns about missing <code> tags, offering to include them.

(I'm a bit tired of unreadable posts here, and all the following edit-considerations and replies)

Cheers Rolf

PS: I'm not sure if this thread better belongs to PM-Discussions.

Update

Other ideas:

(average) line length
code is shorter than regular text
indentation :
text has rarely indented parts
word frequency :
statistics should show significant frequency differences of keywords in text and code
genetic algorithm trained on archive :
downloading old posts to optimize best mix of different metrics
typical starters
shebang, use strict; ...
Conditional_probability / Naive_Bayes_classifier
combining the results of different checks

Interesting links
  • highlight.js
  • SyntaxHighlighter.js
  • naive bayes classification course (Perl)
  • identify-programming-languages-with-source-classifier (Ruby)
  • how-to-detect-programming-language-from-a-string (SO)
  • detecting-programming-language-from-a-snippet (SO)

    In reply to heuristic to detect (perl) code by LanX

    Title:
    Use:  <p> text here (a paragraph) </p>
    and:  <code> code here </code>
    to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.