in reply to Re^6: Multi-CPU when reading STDIN and small tasks
in thread Multi-CPU when reading STDIN and small tasks
how could the processing be sped up?
At the moment you have a triple nested loop over every entry in your hash/subhash/subhash every few seconds. That's a disaster.
If you know that when the timestamp of a record from a node changes, it is the start of a new event, you simply accumulate event data from each node until the timestamps changes, write the current accumulation and replace it with the new record.
Events with multi lines are rolled up as the multi-parts arrive, and are output immediately the first part of a new event is received. You don't even have to check for EOE types, though you could continue to if you preferred.
The possible caveat of this approach is that each event from a given node is only written when the (first line of the) next event is received; thus if a node goes down, its last event may never get written.
If that is unacceptable, then you would need to re-institute a timeout mechanism.
|
|---|
| Replies are listed 'Best First'. | |
|---|---|
|
Re^8: Multi-CPU when reading STDIN and small tasks
by bspencer (Acolyte) on Jan 30, 2017 at 01:27 UTC |