If you really are storing numerical data with numerical keys, an array might serve you better than a hash.

Have you tried profiling your code to make sure that this really is the part that needs to be optimized? I find it hard to believe that in any useful application, storage and retrieval in RAM is the bottleneck.