Re^3: threads::shared vs IPC::Shareable

Id like it in multiple processes due to size of logs. As each preprocesser is processing GBs of logs having each executing in its own process space would be important to take advantage of multiple cores.

Where/what type of storage do these logs files exist on? Because running multiple concurrent file readers can slow things down rather than speed them up if you aren't very careful.

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)

In the absence of evidence, opinion is indistinguishable from prejudice.

Comment on Re^3: threads::shared vs IPC::Shareable

Replies are listed 'Best First'.
Re^4: threads::shared vs IPC::Shareable by Nar (Novice) on Dec 02, 2015 at 21:50 UTC
They are coming in by syslog, I have a perl syslog server listening on a socket. Each preprocesser will access a shared hash to access the logs. As most logs are trash and preprocessed out and the other logs dropped this should keep IO low. ---- I don't throw the logs out in syslog as they are used to preprocess the other logs, but still not relevant for the engine.	[reply]
Re^5: threads::shared vs IPC::Shareable by BrowserUk (Patriarch) on Dec 03, 2015 at 00:27 UTC
Originally you said: Multiple preprocessers will run and preprocess different log files moving the results to a hash then a single processer will pull contents from the hash and delete the element after processing. So, the preprocessors read files and write to a shared hash; and the processor reads the hash. Which somewhat agreed with this description. But now you say: Each preprocesser will access a shared hash to access the logs. The preprocessors read the hash to access the logs. At this point, I've no idea what it is that you are doing, so the best I can say is good luck. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". I knew I was on the right track :) In the absence of evidence, opinion is indistinguishable from prejudice.	[reply]