I am (now) going down the road of poor-mans-threading and letting rsyslog start up the same script based on criteria. It's working well in the small scale POC we have going.

The deduplication is purely from space savings standpoint and had little or no impact on overall searches being performed. Splunk will be ingested in the data and will be performing the searches against the data. That side of this effort is currently handled by another team and I have no or little insight into it apart from being sure I can get them the data in a format which will work for them.