in reply to Persistent File Queue recommendation requested

You could use the standard technique that's used to prevent multiple processes modifying the same file:
  • Comment on Re: Persistent File Queue recommendation requested

Replies are listed 'Best First'.
Re^2: Persistent File Queue recommendation requested
by r1n0 (Beadle) on Jan 27, 2010 at 14:35 UTC
    I tested using a tied array (to file) with what you mentioned. I was locking a shared variable before adding anything to the array and I would lock the same variable before a read. The problem I encountered (and I hope I am wrong) is that Tie::File module seemed to be leaky (memory wise), and before I knew it, my overall script was taking a lot of memory. My script starts with ~20MB mem being used, and after using Tie::File and performing your steps above for about 30 minutes, the script would grow to over 500MB in RSS memory. Does anyone know if Tie::File is leaky?
[reply]
      I've never heard before it being leaky. What makes you think the leak is in Tie::File, and not in your usage? Considering that Tie::File uses a cache, you may end up with each thread having a separate cache; each of them consuming memory (and if each of them has their own view of the file, the threads won't communicate with each other correctly either).

      How are you using Tie::File so that the threads can divide the work correctly?

[reply]
        Probably not the most efficient way to use Tie::File, but I have a subroutine that I use to tie @arrays to files and push or shift the information from the @arrays.
        Different threads are working at the same time, and I am sure I can do a better job of working with the Tie::File. I am glad you said Tie::File is probably not leaky. I use it on other projects and don't have a problem with it, but those projects are not multithreaded.
        In the next few days, I will test something and post what I use for the test. I wrote a very simple script (2 threads) that uses my method for utilizing Tie::File and the script kept growing by 4KB at a time. After I post, I am hoping someone can point out my flaw.

        Thanks!
[reply]

In Section Seekers of Perl Wisdom