hashing it (@array) would take up too much RAM...

A hash is still your best bet though. You could consider a disk based hash like DBM::Deep.