As others said, don't store the result set in memory.

RAM is not going to help necessarily. Windows is a Virtual Memory operating system, it uses pagefile.sys as virtual memory when RAM is over-subscribed. On a 32-bit machine Windows only allows ~ 2GB virtual memory to a user process (there are hacks around that - but don't go there).

Take a look in the Event Log, system and application logs, there may be something in there. However it will almost certainly be some sort of out-of-memory error.