Close. A hash value is a scalar so you need a reference to an array. In fact you want a reference to an array of arrays so that looks like:

use warnings;
use strict;

my %number = (
    23 => [['g1', 23, 24], ['g2', 34, 35]],
    24 => [['g3', 45, 56], ['g4', 36, 48]],
);

print "First group is: $number{23}[0][0]\n";
[download]

Prints:

First group is: g1
[download]

Note the use of () (not {}) for the initialization list for the hash and that the group names need to be quoted.