They are coming in by syslog, I have a perl syslog server listening on a socket. Each preprocesser will access a shared hash to access the logs.
As most logs are trash and preprocessed out and the other logs dropped this should keep IO low.
----
I don't throw the logs out in syslog as they are used to preprocess the other logs, but still not relevant for the engine.