I had same problem at home: 512MB RAM, 512MB swap, 3GHz CPU. I left the computer on (it slurped a lot of swap) for a few hours, and the build succeded without problems. I'm happy Rakudo* user now.
Well thats OK if you want to just compile and use, but if you are exploring the internals the change-build-test cycle takes a lot of time and test of patience.