is this some shell error

The only relevance I can see of a shell is that you could check the ulimit settings.

You don't state which operating system, but I assume some sort of *nix. UNIX is a virtual memory operating system, and RAM is only one of the memory resources. More likely though you are hitting a kernel limit setting. UNIX and Linux have hard and soft limits on memory allocated for user processes and (sometimes) for kernel. You might be hitting one of those limits.

Its a long shot, but have you checked the administrator's log files (like /var/log/messages) for similar errors? They might point you in the right direction.