Re^4: Multi-CPU when reading STDIN and small tasks

I don't think the assumptions are correct. Let me attempt to clarify/correct.

Each event type (these both happen to be SYSCALL) with each type having it's own definition in both the number of lines making up the event and the content of each. Most event types contain an End of Event marker (the type=EOE) indicating that all of the data for that event has been sent. This is not 100% true as at least one event is only a single line with no EOE (one use of the expiry). Each node will have any and all of the different event types
The multi-line entry (I may interchange this with "event" but we are referring to the same thing) is complete when the EOE record has been received or ?? for the event types which do not contain an EOE line. I haven't attempted to look up all of the event types and their definitions as I assumed that would be more heavy on the processing side to match on more patterns.

Additional maybe-useful information:

The unique key is a combination of the node and the part after the colon in the msg=audit() section.
I'm trying to get from the start to this state (below) as quickly as possible and if constrained to a single thread, with the least amount of CPU usage. The "dedupe" regex seems to account for roughly 50% of the time in the loose testing I've performed. It also contributes to some of the CPU usage but I feel it's less than 50% of the CPU from observation of before/after adding it.

If this helps, this is the final result of the processing (The new line added for readability and to signify that a new line is being started):

node=xxxxxxxxxx type=SYSCALL msg=audit(1485583201.776:5485082): arch=c000003e syscall=82 per=400000 success=yes exit=0 a0=7fc164006990 a1=7fc164006b70 a2=7fc164006b70 a3=7fc230853278 items=4 ppid=xxxxx pid=xxxxx auid=xxxxx uid=xxxxx gid=xxxxx euid=xxxxx suid=xxxxx fsuid=xxxxx egid=xxxxx sgid=xxxxx fsgid=xxxxx tty=(none) ses=4294967295 comm="somecommand" exe="/full/path/to/somecommand" key="delete" type=CWD  cwd="/another/cwd" type=PATH item=0 name="arg-data-0" inode=268805 dev=fd:14 mode=040740 ouid=xxxxx ogid=xxxxx rdev=00:00 nametype=PARENT item=1 name="arg-data-1" item=2 name="arg-data-2" inode=269256 mode=0100640 nametype=DELETE item=3 name="arg-data-3" nametype=CREATE

node=aaaaaaaaaa type=SYSCALL msg=audit(1485583203.459:5485148): arch=c000003e syscall=59 success=no exit=-2 a0=7f30b9d87149 a1=7f30b9d86860 a2=7f30b9d86bd8 a3=7f30b9d9c8c0 items=1 ppid=xxxxx pid=xxxxx auid=xxxxx uid=xxxxx gid=xxxxx euid=xxxxx suid=xxxxx fsuid=xxxxx egid=xxxxx sgid=xxxxx fsgid=xxxxx tty=(none) ses=16439 comm="command" exe="/bin/ksh93" key="cmdlineExecution" type=CWD  cwd="/a/cwd" type=PATH item=0 name="/etc/uname" nametype=UNKNOWN

Comment on Re^4: Multi-CPU when reading STDIN and small tasks

Replies are listed 'Best First'.
Re^5: Multi-CPU when reading STDIN and small tasks by BrowserUk (Patriarch) on Jan 29, 2017 at 01:49 UTC
Hm. According to Red Hat Audit log files, assuming you and they are talking about the same thing, my assumptions are correct. If they are, your processing can be sped up considerably by utilising them. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". The enemy of (IT) success is complexity. In the absence of evidence, opinion is indistinguishable from prejudice.	[reply]
Re^6: Multi-CPU when reading STDIN and small tasks by bspencer (Acolyte) on Jan 29, 2017 at 12:29 UTC
You are correct as that is the data which is being processed however their example presented there is only one sample and there are variations even within the event types. I haven't realized some of this until now as well and points out the samples I gave were maybe not the best. Their SYSCALL example: 3 records (SYSCALL, CWD and PATH) The two samples I gave:3 records (SYSCALL, CWD and PATH) Other SYSCALL data: 5 records (SYSCALL, EXECVE, CWD, PATH and PATH) Other Types (LOGIN): 1 record (LOGIN) There is a list record types but it does not explain which records relate to which events. I believe that the records always appear in the same order within the event though. If there was consistency in the events and records (if that is required), how could the processing be sped up?	[reply]
Re^7: Multi-CPU when reading STDIN and small tasks by BrowserUk (Patriarch) on Jan 29, 2017 at 18:13 UTC
how could the processing be sped up? At the moment you have a triple nested loop over every entry in your hash/subhash/subhash every few seconds. That's a disaster. If you know that when the timestamp of a record from a node changes, it is the start of a new event, you simply accumulate event data from each node until the timestamps changes, write the current accumulation and replace it with the new record. Events with multi lines are rolled up as the multi-parts arrive, and are output immediately the first part of a new event is received. You don't even have to check for EOE types, though you could continue to if you preferred. The possible caveat of this approach is that each event from a given node is only written when the (first line of the) next event is received; thus if a node goes down, its last event may never get written. If that is unacceptable, then you would need to re-institute a timeout mechanism. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". The enemy of (IT) success is complexity. In the absence of evidence, opinion is indistinguishable from prejudice.	[reply]
Re^8: Multi-CPU when reading STDIN and small tasks by bspencer (Acolyte) on Jan 30, 2017 at 01:27 UTC