What are the performance impacts of this?

It would have to be measured, i.e. bench-marked, with real data.

However, my gut feeling is that removing one level of nested-ness is likely to speed up things a bit, but probably not by a large margin. I doubt that you really care about the difference for what you're doing. So, don't worry too much about performance, unless you really have to.

The hash solution (especially with concatenated keys) is very likely to use far less memory, at least with sparse data. Suppose you've got only one data point with coordinates (800, 1200). With an array of arrays, you have to allocate essentially 800 * 1200 array slots, that's quite a lot or memory for just one data piece. But with a hash you need to allocate only one or two hash entries; even considering that a hash entry uses more memory than an array entry, there is a significant win here.