The ^ sign tells the regex to trim only the first chacaracter of a line, as the $ sign tells to trim the last one. If you want to get rid of all double quotes you should remove it:

$line =~ s/\"//g;

Type perldoc perlre for more information.

Regards,