in reply to Re: how to split huge file reading into multiple threads
in thread how to split huge file reading into multiple threads

Thanks!

Yes. even I am under the impression that perl as designed for pattern extraction and report; would be fast and reliable for text processing.

Please see my (above) reply to:"AR" of dated:"Aug 30, 2011 at 09:05 UTC" what my code does.

Thank you for extending the hand. However, as of now, I completely think that there is not really much I am doing (as in processing the records) that would consume the time. I dont want other monks to get mis-directed by pasting the code.

  • Comment on Re^2: how to split huge file reading into multiple threads

Replies are listed 'Best First'.
Re^3: how to split huge file reading into multiple threads
by BrowserUk (Patriarch) on Aug 30, 2011 at 09:57 UTC
    I dont want other monks to get mis-directed by pasting the code.

    Let "other monks" look after themselves.

    If your code is taking 2 1/2 hours to process 20 million records against 600 records stored in a hash, then it is your code that has problems. Should we try and guess what mistakes you are making?

    Are you for instance, treating the hash as an array? Or re-opening the output files for every record you write?

    Post the code and we won't have to make such guesses.

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
      Hey I posted the code. Can you please suggest now.
[reply]

        If you mean the code you posted in 923771, it doesn't even compile:

        Unmatched right curly bracket at - line 107, at end of line
syntax error at - line 107, near "}"
Unmatched right curly bracket at - line 108, at end of line
Unmatched right curly bracket at - line 109, at end of line
Execution of - aborted due to compilation errors.

        [download]

        And even if it did compile, running it would do nothing as all it contains are two subroutines, that are never called.

        If you have actually run a program that takes 2 1/2 hours to process "millions of records", then why haven't you posted that code?

        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]

In Section Seekers of Perl Wisdom