If you want to avoid potential issues w/ regex metacharacters, you can use a set of hash keys to track what's been seen and rebuild the regex once for each character:

#!/usr/bin/perl 
use strict;
use warnings;

my %char_hash = ();
$char_hash{ chr($_) } = undef foreach (33 .. 127);
my $chars = join "", keys %char_hash;
my $regex = "([\Q$chars\E])";

while (<DATA>) {
    while (/$regex/g) {
        delete $char_hash{$1};
        $chars = join "", keys %char_hash;
        $regex = "([\Q$chars\E])";
    }
}
my @good_array = keys %char_hash;
print @good_array;

__DATA__
!"#$%&'()*+,-./01234567
89:;<=>?@ABCDE
FGHIJKLMOPQRSTUVWXYZ[\]^_`abcdefghijklmnop
qrstuvwxyz{|}~
[download]

though I feel like there must be a simpler way of implementing this approach.