in reply to collective unconcious (about shared memory...)

On most modern versions of Unix, all new processes will initially share all (or nearly all) of their memory with their parent (each new process gets its own new list of page table entries that define how to map virtual memory address ranges into real memory and/or swap space pages). After that, when any of the related processes writes to a shared page of memory, a fault will occur which will cause that page of memory to be copied to a page that is local to the writing process.

So you have to let your fork()ed processes run for a while before you can get a feel for how much memory they will manage to continue to share.

Since IO::Socket::SSL eventually uses some C code, you should be sure to put that code into a shared library. That way that chunk of memory will always remain shared between all processes (even if they aren't related) since the shared library is loaded using mmap (or an equivalent).

Perl code compiles to an opcode tree. I'm curious if Perl manages to leave large chunks of this tree as read-only so that related processes can continue to share that memory.

        - tye (but my friends call me "Tye")
  • Comment on (tye)Re: collective unconcious (about shared memory...)

Replies are listed 'Best First'.
Re: (tye)Re: collective unconcious (about shared memory...)
by dws (Chancellor) on Apr 17, 2001 at 08:15 UTC
    Perl code compiles to an opcode tree. I'm curious if Perl manages to leave large chunks of this tree as read-only so that related processes can continue to share that memory.

    My knowledge of *nix kernel memory management is stale at this point, so take this with a grain of salt.

    It used to be the case that the read-only pages were so marked in the executable file. The executable also contained some number of writeable data pages, and info for creating a "blank static storage" section to house uninitialized structures. Additional storage (e.g., allocated by malloc()) grew upwards from writable storage via sbrk(), and the stack grew downwards from high memory.

    Perl process build their opcode trees in writeable space. I'm not aware of a mechanism to mark writeable pages as read-only at runtime.

    However, depending on how fork() is implemented, writeable pages from the parent process may shared with the child process using a "copy on write" scheme, in which pages are shared until one of the process tries writes into the page, at which point a copy is made for each process. Such a scheme would work well with Perl, allowing the opcode tree to be shared.

    Now, will somebody please update me by telling me how this is wrong?

[reply]
[d/l]
[select]

      What I was describing was "copy on write" (I probably should have at least mentioned that term, eh?). I must have done a poor job of describing it if you didn't recognize it. Sorry.

      My question was whether code that is compiled before the fork() would remain shared (for very long) after the fork() or whether Perl updates things (like reference counts or state information) in the opcode tree itself such that Perl running the code causes the opcode tree to be written to and that memory to become unshared.

      I recall this coming up a long time ago and vaguely recall that the opcode tree was written to and someone thinking about trying to change that. I wonder if that memory is at all accurate and if any changes were made.

      BTW, your description of memory architecture matches my understanding of the common layout. Perhaps one missing piece is how mmap() fits in. mmap() was a great idea that unified how the page file (a.k.a. swap space) worked with shared memory and I/O buffers.

              - tye (but my friends call me "Tye")
[reply]
        Your previous memory was probably the thread starting at this post. I have no idea whether anything happened since that conversation.
[reply]
Re: (tye)Re: collective unconcious (about shared memory...)
by dshahin (Pilgrim) on Apr 17, 2001 at 21:47 UTC
    >Since IO::Socket::SSL eventually uses some C code, you should be sure to put that code into a shared library. That way that chunk of memory
    >will always remain shared between all processes (even if they aren't related) since the shared library is loaded using mmap (or an equivalent).

    I'm not sure what you mean by this.

    So parents and children share memory, but what about invocations of the Perl interpreter?
    That is, if I invoke the same program from different shells, do they share memory? They must use the same shared libs, right?

    dshahin

[reply]

      Yes, one of the points behind shared libraries is that the read-only sections of those libraries get shared by all processes that use the same library (even if the processes aren't related).

      The parent and child [after fork() but before exec()] have the potential to share a lot more memory.

              - tye (but my friends call me "Tye")
[reply]

In Section Seekers of Perl Wisdom