in reply to Re^2: Multi-CPU when reading STDIN and small tasks
in thread Multi-CPU when reading STDIN and small tasks

I'm glad to hear that you are seeing some performance increases! This 2-3x is in the range of what I expected with index instead of regex, but that evidently is still "not enough":

Without the garbage collection this still is not realistic but helps in being able to judge the impact of certain choices which is what I have been doing a lot of over the last couple of days.

What performance benchmark do you think needs to be met in order for the system as a whole to work? Since you have excluded the time-out code for the moment and we've tweaked a number of issues, there isn't a whole lot of "meat" left on these bones!

I don't see any super easy miraculous 10x (order of magnitude) solution. Even writing this thing in C is maybe just another 2-3x. From "reading between the lines", it sounds like you would like to do even more processing than the code that we've been benchmarking?

Backing up a bit about the requirements... how does the output from your hundreds of servers come to be merged into a single pipe? Is there some way to distribute the load further "upstream" into multiple "fire hoses" instead of just a single one?
Is it ok if Server123's data is on a separate machine from Server789's? It sounds to me like a server process model is more appropriate than threads because this is sounding like you will wind up needing multiple machines. That kind of approach can yield a 10x type of performance increase and be scalable.

Of course of interest is what is driving your requirements to begin with? What is the "user end product" result? I mean so we have collected all the lines for a single node/time/event into a single line, so what? Why is that a requirement and why is that helpful? Maybe there is a way to do the processing of whatever "end result" you desire without this very high performance program? I don't know, but this is an obvious question.

Update: Another thought about your benchmark,
$ time cat audit.log|./auditd-linux-orig.pl >/dev/null
This running of cat and piping into auditd-linux-orig.pl and re-directing shell output could potentially have some performance impact. Out of curiosity, is there any difference if auditd-linux-orig.pl opens a file handle for read from audit.log and a file handle for write to /dev/null? Instead of using the shell re-direction? Of course there is also a small difference included in your benchmark for Perl to load and compile. I am currently using Windows and I'm not sure if any measurement that I made would be applicable to your system.

Replies are listed 'Best First'.
Re^4: Multi-CPU when reading STDIN and small tasks
by bspencer (Acolyte) on Feb 03, 2017 at 16:48 UTC

    Distracted over the last few days actually working on some of the items you mentioned (before seeing your comments).

    What performance benchmark do you think needs to be met in order for the system as a whole to work?
    Each machine is on average sending 170 events / second in the little POC we are working through. Ideally I would have liked to see a single instance handle around 100K lines to reduce the number of other workarounds required. Using the index method (without nothing else such as writing/processing) peaked at about 60K which would be workable but then again, it's not actually processing as it needs to.

    it sounds like you would like to do even more processing than the code that we've been benchmarking?
    The obvious thing which the code wasn't doing in these tests was writing to files. It was simply displaying to STDOUT instead to allow for format confirmation as needed. This is why STDOUT was redirected to /dev/null in the tests. The other extra processing, the removal of duplicate fields, was accounted for in the tests.

    how does the output from your hundreds of servers come to be merged into a single pipe?
    Syslog basically. Each server -> central syslog servers -> STDOUT piped to script -> written to disk -> ingested into something which can't deal with the auditd format

    Is there some way to distribute the load further "upstream" into multiple "fire hoses" instead of just a single one?
    There is and it is the path I've started going down based on last weekend's exploration. Until 10's of conditions are introduced this seems to be workable as a method of spreading the load: 

    Server ->                   Script Instance A (based on condition A)
Server ->  Syslog server -> Script Instance B (based on condition B)
server ->                   Script Instance C (based on condition C)
    One of the requirements is NOT that there is a single file in the end so A, B and C will each create their own file in order to avoid locking/contention between the different "threads". I think, based on an attribute in the server name, I think this may be a workable solution and the number of servers per "thread" will work at least today. I'm still working on this setup to confirm it.

    Of course of interest is what is driving your requirements to begin with?
    The end requirement is to have the auditd data from all of the servers but in a "flattened" format in files so that they can be read into something which will analyze them. Because of the amount of data involved and about a reduction of space required by about 30%, we added the deduplication of data in each of the single lines events.

    Out of curiosity, is there any difference if auditd-linux-orig.pl opens a file handle for read from audit.log and a file handle for write to /dev/null?
    This is not an item I tried. I could have just as easily commented out the print statement which is the only need for the redirection. In actual use the script is reading from STDIN (I like using cat to simulate that) and writing to files. STDOUT is never used.

[reply]
      Ok, now I understand the performance requirements better.

      Doubling the performance from 60K to 120K lines/sec with your current single process would be possible albeit with some C code. But that still wouldn't do all that you want. I predict that I could code $singleline=~s/((\S+)\s?)/$count{$2}++ ? '' : $1/eg; much more efficiently in ASM rather than in C because there are certain instructions that are difficult for the C compilier to even use. If this was an embedded hardware board application, it would be worth the effort. But here, I think not! I believe you are better served with a pure Perl application

      I think you are on the right direction to distribute this incoming "firehose of data" between multiple processing entities. Right now it appears that you are thinking about one program with multiple threads. I would be thinking of multiple instances of a single threaded process with a "router" process. Let the OS assign these processes to different machine cores. I don't see any requirement for these processes to communicate with each other or share information. A consideration could be how easy it is to just add an additional machine when the load increases?

      Leave your final "print" in the benchmark. That does all the work and it does go to STDOUT, it just result gets re-directed to the "bit bucket".

      I am still curious as to what this analysis program does with this massive amount of data? It seems that some kind of "front-end" to this thing might be possible? Extract perhaps a time window, perhaps all data from Server X from the main log file that is then analyzed in non-real time. It seems to me that the processing power of super fast concatenation of lines and the compression of the data by 30% due to "dupes" must be minuscule to the overall effort of the analysis program? Aside from reducing the storage required, it is not clear how much this will help the "final end result"?

[reply]
[d/l]

        I am (now) going down the road of poor-mans-threading and letting rsyslog start up the same script based on criteria. It's working well in the small scale POC we have going.

        The deduplication is purely from space savings standpoint and had little or no impact on overall searches being performed. Splunk will be ingested in the data and will be performing the searches against the data. That side of this effort is currently handled by another team and I have no or little insight into it apart from being sure I can get them the data in a format which will work for them.

[reply]

In Section Seekers of Perl Wisdom