in reply to Re^2: how to split huge file reading into multiple threads
in thread how to split huge file reading into multiple threads

It dawned on me there is another way to collect the output from the threads. You can open a filehandle in the main thread, pass it's fileno to the thread, then let the thread write to the dup'd filehandle. See [threads] Open a file in one thread and allow others to write to it for the technique.

Anyways, you could open 1 filehandle for each thread, for that thread to report results back to the main thread. Pass the fileno of that filehandle to each thread at creation time. In the main thread, setup an IO::Select object to watch all the filehandles. Have the main thread open a private filehandle for the final output file, and as IO::Select reads the various data from each thread, it writes the output to the output file.

This would allow the threads to write without worrying about locking, while the main thread's select loop would actually handle the writing, and possibly sorting, the data out to file.

I don't know how it would work speedwise, as select will block if one thread reports alot of data, but this might be minimized by using large filehandle buffers.

That is what I would try first.

I'm not really a human, but I play one on earth.
Old Perl Programmer Haiku ................... flash japh
  • Comment on Re^3: how to split huge file reading into multiple threads

Replies are listed 'Best First'.
Re^4: how to split huge file reading into multiple threads
by BrowserUk (Patriarch) on Aug 30, 2011 at 23:08 UTC

    Could you explain a little more about how you envisage this working please?

[reply]
      Foreach chunk of the split file, a thread will be created. In the thread creation loop, each thread will get passed the chunk filename, and a unique fileno. The fileno will be derived from a rw (+>) filehandle created in the loop, 1 for each thread. In the main thread, each of those filehandles would be added to an IO::Select object, and after the main thread's worker-thread-creation loop is finished, the main thread would sit in a loop watching the IO::Select object. The threads, would dup the fileno's for writing, and write their output there.

      The idea would probably work also for forked worker processes, but you would need to pass the $pid of the parent process as well as a fileno; since filehandles used by the same owner are writable by all processes of that owner.

      The IO::Select loop in the main thread would be similar in setup, to a socket-watch program. As the data comes in to $select->can_read, it will read the data( preferably with sysread in huge chunks), and just copied to an output filehandle.

      A few points the OP would have to watch are

      1.Making sure the original huge file split dosn't split in the middle of a line, rendering a few records broken.

      2. Making sure that IO::Select dosn't clog up and slowdown the output of some threads, by 1 overly aggressive thread outputting too much and hogging the Select object. One possible solution would be to use the largest filehandle buffers possible on the platform, so slower threads can keep outputting to the buffers, if one thread's output becomes very heavy.

      The code should be fairly straightforward, and possibly someone as agile with thread code as you, could whip out some code quickly. For me, it would take all morning, and I prefer f'ing off. :-)

      I'm not really a human, but I play one on earth.
      Old Perl Programmer Haiku ................... flash japh
[reply]
        you, could whip out some code quickly.

        That is my intent, to get something working and see how it fairs. But I'm still not getting a clear picture from your description. A bit confused.

        The file handles the main thread is selecting on, are these the same ones that your passed their filenos to threads for duping? Does that mean that the main thread is can-reading on rw dups of teh same files that the threads are writing to?

        IF you could knock up a little code to show the flow -- it doesn't have to work, I can knock it into shape -- then it would probably be quicker than me asking 20 questions trying to work out which filehandles do what :)

        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

In Section Seekers of Perl Wisdom