"This isn't working." is a totally inadequate problem description.

Both " and, the less cryptic, " are HTML character entity references for a double quote ("). Your regex works for me:

$ perl -Mstrict -Mwarnings -E 'my $x = qq{X\xE2\x80\x9CX}; $x =~ s/[\x
+E2][\x80][\x9C]/"/g; say $x'
X"X
$ perl -Mstrict -Mwarnings -E 'my $x = qq{X\xE2\x80\x9CX}; $x =~ s/[\x
+E2][\x80][\x9C]/"/g; say $x'
X"X
[download]

You can also substitute a literal double quote:

$ perl -Mstrict -Mwarnings -E 'my $x = qq{X\xE2\x80\x9CX}; $x =~ s/[\x
+E2][\x80][\x9C]/"/g; say $x'
X"X
[download]

Please read: How do I post a question effectively?