If this is not homework, Text::CSV_XS is your friend. (If this is homework, Text::CSV_XS will be your friend once you're out in the Real World.) For your test input file:

c:\@Work\Perl\monks\oop11999>perl -e
"use warnings;
 use strict;
 ;;
 use Text::CSV_XS;
 ;;
 use Data::Dump qw(dd);
 ;;
 my $csv = Text::CSV_XS->new ({
   sep_char => ' ',
   })
   or die qq{Cannot use CSV: }, Text::CSV_XS->error_diag;
 ;;
 open my $fh, '<', 'test.csv' or die qq{opening test.csv: $!};
 ;;
 my %letters;
 while (my $row = $csv->getline($fh)) {
   my ($letter, @numbers) = @$row;
   push @{ $letters{$letter} }, @numbers;
   }
 $csv->eof or $csv->error_diag;
 close $fh or die qq{closing test.csv: $!};
 ;;
 dd \%letters;
 ;;
 for my $letter (sort keys %letters) {
   print qq{$letter $_ \n} for @{ $letters{$letter} };
   }
"
{ A => [1], B => [3, 6, 2], C => [3, 4, 6, 2, 10] }
A 1
B 3
B 6
B 2
C 3
C 4
C 6
C 2
C 10
[download]

(The  dd \%letters; statement is just for debug and illustration.)