First off, the hash you're building is ... wonky. Why not just use an array of arrays instead of a hash of hashes? All your keys are numbers ... that indicates you really want an array, not a hash.

Second, unless your CSV format is vastly different than the ones I'm used to, your s/"//g; will LOSE information, badly. Like, it'd be trivial to make a correct CSV file that would make your code break.

Much better is to let a CPAN module do this for you. I like tilly's Text::xSV best, but Text::CSV_XS is perfectly acceptable. (Text::CSV isn't feature-complete.)

use Text::xSV;

sub build_hash {
    my ($file_) = @_;

    my $reader = Text::xSV->new();
    $reader->open_file( $file_ );

    my @result;
    while ( my $row = $reader->get_row() ) {
        push @result, $row;
    }

    return @result;
}
[download]

If you absolutely have to have a hash of hashes, then you could do something like:

use Text::xSV;

sub build_hash {
    my ($file_) = @_;

    my $reader = Text::xSV->new();
    $reader->open_file( $file_ );

    my %result;
    my $line = 0;
    while ( my $row = $reader->get_row() ) {
#        @{ $result{ ++$line } }{ 1 .. scalar(@$row) } = @$row;

        $line++;
        for my $i ( 1 .. @$row ) {
            $result{$line}{$i} = $row->[ $i-1 ]
        }
    }

    return %result;
}
[download]

That commented-out wonky line is a hash-slice. It's exactly equivalent (but faster) than the 4 lines below.