I don't think you can have it both ways. You either need to work around having a reference or live with the major memory requirement. I'm not an XS-guru, but it appears Data::Swap just does the reference stuff for you.