The simple fix is to replace all occurrences of commas in your regular expression by escaped backslashes, not just the first, i.e:

s/\\(?:"[^"\r\n]*"|[^\\\r\n]*)$//m;

I would also point out that $_ =~ is unnecessary since s/// implicitly binds to the magic variable $_ when you don't provide an explicit binding.

And you should probably be using Text::CSV with the pure Perl implementation Text::CSV_PP if you don't want to deal with XS on windows. And barring that, split is usually a better substitute.