There is no way to answer your question on the basis of the (lack of) information you've provided.

The first questions I have are:

• Why are you sharing a hash?
• What are the keys to the hash?
• Does the information being stored in the value associated with each key come from a single preprocessor; or is it accumulated from multiple preprocessors?

  If it is from a single preprocessor, why does it need to be shared through a hash?

  If it is accumulated from multiple preprocessors; how will the processing app know when any given key is ready for post processing?

And many more question arise depending upon the answers to those.

Basically, until you describe what information is being passed from the preprocessors to the processor; how that information is keyed; and how it is derived and/or accumlated; any response to your question is an uneducated guess.

use strict;
use warnings;

my %hash;

while(<DATA>){
    my ($key, $value) = split(/\s+/, $_);
    $hash{$key} = $value;
}

foreach(keys %hash){
    if ($_ =~ /this|is|a|test/){
        print "found string \"$_\" and its value is \"$hash{$_}\"\n";
    }
}

__DATA__
this 1
is 2
a 3
test 4
[download]

This would read a logfile and add all of its contents to the hash. Then you can do functions foreach key or value. Honestly, I wouldnt even use a hash or an array, I would just loop through the logfile and do functions foreach line. I cannot comment about threads as I never had any situation that I absolutely needed them myself.

Cache::FastMmap might help. I use it here

What is the purpose of the hash here? Do you need to build some lookup index? Sort or categorize?

Natural fit for a preprocessor is to function as a filter. Keep state but don't slurp the data. You can pipe it and stream it. But if complicated queries are required, then a database model might be more suitable.

Id like it in multiple processes due to size of logs. As each preprocesser is processing GBs of logs having each executing in its own process space would be important to take advantage of multiple cores.

It uses hashs so as multiple preprocessers write, the processer can execute in order of datetime.

Each preprocesser would write to the same hash and the engine would process that hash.

Id like it in multiple processes due to size of logs. As each preprocesser is processing GBs of logs having each executing in its own process space would be important to take advantage of multiple cores.

Where/what type of storage do these logs files exist on? Because running multiple concurrent file readers can slow things down rather than speed them up if you aren't very careful.

---

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)

In the absence of evidence, opinion is indistinguishable from prejudice.