It sounds like your idea should work. There's a module that might help you experiment with av_store and hv_store before descending to XS code: Array::RefElem.