Any reason not to just use a lexical variable for the handle? You can also get rid of the $1 if since push imposes list context on the regex anyway.

my @COLOR;
{
    open my $fh, "<", "/usr/lib/X11/rgb.txt" or last;
    push @COLOR, /\d+\s+\d+\s+\d+\s+(\S+)/ while <$fh>;
}
[download]

Makeshifts last the longest.