Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options
 
PerlMonks  

Re^2: threads::shared vs IPC::Shareable

by Nar (Novice)
on Dec 02, 2015 at 20:11 UTC ( [id://1149213]=note: print w/replies, xml ) Need Help??


in reply to Re: threads::shared vs IPC::Shareable
in thread threads::shared vs IPC::Shareable

Id like it in multiple processes due to size of logs. As each preprocesser is processing GBs of logs having each executing in its own process space would be important to take advantage of multiple cores.

It uses hashs so as multiple preprocessers write, the processer can execute in order of datetime.

Each preprocesser would write to the same hash and the engine would process that hash.

Replies are listed 'Best First'.
Re^3: threads::shared vs IPC::Shareable
by BrowserUk (Patriarch) on Dec 02, 2015 at 20:24 UTC
    Id like it in multiple processes due to size of logs. As each preprocesser is processing GBs of logs having each executing in its own process space would be important to take advantage of multiple cores.

    Where/what type of storage do these logs files exist on? Because running multiple concurrent file readers can slow things down rather than speed them up if you aren't very careful.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)
    In the absence of evidence, opinion is indistinguishable from prejudice.
      They are coming in by syslog, I have a perl syslog server listening on a socket. Each preprocesser will access a shared hash to access the logs. As most logs are trash and preprocessed out and the other logs dropped this should keep IO low. ---- I don't throw the logs out in syslog as they are used to preprocess the other logs, but still not relevant for the engine.

        Originally you said:

        Multiple preprocessers will run and preprocess different log files moving the results to a hash then a single processer will pull contents from the hash and delete the element after processing.

        So, the preprocessors read files and write to a shared hash; and the processor reads the hash. Which somewhat agreed with this description.

        But now you say:

        Each preprocesser will access a shared hash to access the logs.

        The preprocessors read the hash to access the logs.

        At this point, I've no idea what it is that you are doing, so the best I can say is good luck.


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)
        In the absence of evidence, opinion is indistinguishable from prejudice.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1149213]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others examining the Monastery: (6)
As of 2024-03-28 23:23 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found