in reply to threads::shared vs IPC::Shareable

What is the purpose of the hash here? Do you need to build some lookup index? Sort or categorize?

Natural fit for a preprocessor is to function as a filter. Keep state but don't slurp the data. You can pipe it and stream it. But if complicated queries are required, then a database model might be more suitable.

Replies are listed 'Best First'.
Re^2: threads::shared vs IPC::Shareable
by Nar (Novice) on Dec 02, 2015 at 20:11 UTC
    Id like it in multiple processes due to size of logs. As each preprocesser is processing GBs of logs having each executing in its own process space would be important to take advantage of multiple cores.

    It uses hashs so as multiple preprocessers write, the processer can execute in order of datetime.

    Each preprocesser would write to the same hash and the engine would process that hash.
[reply]
      Id like it in multiple processes due to size of logs. As each preprocesser is processing GBs of logs having each executing in its own process space would be important to take advantage of multiple cores.

      Where/what type of storage do these logs files exist on? Because running multiple concurrent file readers can slow things down rather than speed them up if you aren't very careful.

      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
        They are coming in by syslog, I have a perl syslog server listening on a socket. Each preprocesser will access a shared hash to access the logs. As most logs are trash and preprocessed out and the other logs dropped this should keep IO low. ---- I don't throw the logs out in syslog as they are used to preprocess the other logs, but still not relevant for the engine.
[reply]

In Section Seekers of Perl Wisdom