Databases are effective in this case because they understand the importance of streaming data to/from disk rather than using it as RAM. As for why, see Re^2: Memory Efficient Alternatives to Hash of Array. However it is fairly easy for you to acquire the same knowledge, which basically comes down to knowing to sort then process. Armed with that knowledge, the database slows me down slightly for simple tasks. For complex tasks, the database saves some logic but may come up with an unworkable query plan. I'll therefore give it a try, but push comes to shove I'm willing to go out of the database because I know that I can make it work, and sometimes the database simply won't.