In a nutshell, use a 64-bit operating system (and machine). Operating systems like Windows and Linux do not allow direct access to RAM (unless you are part of the kernel, or a device driver). Instead they use Virtual Memory.
The program needs to access the memory area using an address, and the largest that can be held in a 32-bit word is 4GB.
So where did the other GB go?
That is reserved for kernel.
Kernel uses a Gig?
No, that address range is reserved for kernel use, it doesn't use anything like that.
A 64-bit architecture will, in theory allow an address space of up to 16 Exabytes. | [reply] |
Hmm. If my Perl is running at 3GB, Kernel reserves 1GB, what does this mean for all the other processes, such as Nautilus and System Monitor, which are also running? | [reply] |
It means they are also limited to 3 gigs maximum. The split is not 1-gig for the kernel and 3-gigs for processes, it is 1-gig for the kernel and 3-gigs PER process.
The problem is that a 32-bit address space only gives you 4 gigs of addressable memory, so the processor has an "interpreter" called the Translation Lookaside Buffer which maps virtual addresses into physical addresses. So your perl application sees the system as having 4 gigs of memory, 1 of which is for the kernel and 3 gigs for itself. Other processes see the same thing, but the 3 gigs they are able to address may not be the same 3 gigs of physical memory that other processes see. The upside to this arrangment is that it lets you use more than 4 gigs of RAM in a 32-bit machine. The downside is that the TLB must be flushed when a different process is swapped onto the processor because the addresses that were valid for one process may not be the same addresses for another process.
If memory is the only problem, besides the option to go to 64-bit hardware you could also look at getting (or building) a 4-gig/4-gig split kernel. Red Hat calls this kernel the 'hugemem' kernel, other distros probably have something similar. Using this kernel the kernel gets 4 gigs of address space and processes get 4 gigs each. The downside to this is that the TLB can still only deal with 4 gigs of address space, so with the 3/1 split kernel you have to flush the TLB every time you switch processes, with the 4/4 split kernel you also have to flush the TLB every time you switch from kernel mode to user mode or vice-versa. How much of an impact this has on performance varies (mostly depending on how much I/O your application does), but taking a 60% performance hit for using a 4/4 split kernel is not uncommon.
| We're not surrounded, we're in a target-rich environment! |
|---|
| [reply] |
Of course the other option is to split the workload, if you have 18 gigs of RAM available and the application is able to split the work up among more than one process, then you can have 6 processes using 3 gigs of RAM each and use up all of that 18 gigs of RAM for your workload.
| We're not surrounded, we're in a target-rich environment! |
|---|
| [reply] |
