Try this:

#!/usr/bin/perl
use strict;use warnings;

while (my $line=<DATA>) {
    $line=~s/(?<!\\)#.*//;
    print $line;
    }

__DATA__
\#Not a comment
#Comment
\#Not a comment  #But this is a comment
arbitrary text   #Comment
arbitrary text   \#Not a comment
arbitrary text   \#Not a comment  #But this is a comment
arbitrary text   #Comment         \#This is part of the comment
[download]

Prints:

\#Not a comment

\#Not a comment
arbitrary text
arbitrary text   \#Not a comment
arbitrary text   \#Not a comment
arbitrary text
[download]

The '(?<!' construct looks backward to ensure that the escape character is not preceeding the comment character.