What you're looking for is probably more like:

while (<>) {
  chomp;
  my ($key, @values) = split /,/, $_;
  push @{$numbers{$key}}, [ @values ];
}
[download]

When playing with complex data structures, perldsc and Data::Dumper are your friend.