If I had to point a finger at the most significant factor reducing your performance, I would be tempted to point at your use of system pipes. System pipes have a maximum buffer size, and when this buffer is filled, the producer process must be swapped out. On the other end of the pipe, the same remains true as well. The Perl program (the consumer) can only read one pipe buffer size (often 4096 bytes or 8192 bytes on Unix) before it needs to be swapped out. For very large amounts of data, the constant rescheduling necessary, along with the expectation that other processes on the machine are vying for the same resources, make system pipes inefficient for this solution.

If this is the case, there are a few alternatives you might consider. The first, is that tcpdump could write bytes to a real file, and not a pipe. This would allow tcpdump to pump as much data as it could into the system. In the most optimal solution, the bytes would be written to a file system based on virtual memory, or that used deferred writes, such that the tcpdump write() system calls succeed quickly, and the data is then immediately available to other processes. Your Perl script would then perform a 'read behind' technique that would read() from the file until EOF is encountered. At EOF, a system call such as yield(), poll(), or select() should be executed to yield the processor back to the tcpdump process. When the Perl script is scheduled for execution again, it should read until EOF again. This approach gives you an effectively limitless buffer size, as opposed to the system pipe approach that provides only a fixed (small) buffer size. Of course, the situation of the file becoming too large for the file system may be a consideration.

Otherwise, your only solution would be to hack tcpdump, or find an alternative program than tcpdump, that would invoke the Perl inlines inline, or that would transfer the data more efficiency, such as using a large shared memory segment.

Well, if you are going to write the output to disk file, the program will even be slower, as disks are orders of a magnitude slower than memory. If you are writing to memory, you quickly have a problem because the large amounts of data tcpdump is writing will quickly fill up large chuncks of memory. If it's indeed the buffer size that is the problem, you're better off enlarging the buffer size, by tweaking the OS.

But I expect that this is one of the cases one would prefer to use C instead of Perl.

Abigail

You should note that I suggested the use of a file system that performed deferred writes, or that are based from virtual memory, and not a real disk. In any case, the pages just written are still likely to be in RAM, meaning that the write ahead is to RAM, and the read behind is from RAM. The real benefit of using a true file, is that the system can write the data to the file at its leisure, marking pages as clean and ready for reclaimation as necessary. Using a shared memory segment, for example, requires that the processes perform their own scheduling to ensure that the producer does not over-write pages not yet read by the consumer (during a fast network burst, for example). Using a file lets the kernel do this magic for us.

if you're not keen on writing your own look into flow-tools and fprobe. fprobe uses libpcap (just like tcpdump) and exports netflow format files (just like the bigboy routers). you can then use flow-tools (and lot's of other tools) to give reports and graphs.

if you stick with tcpdump you can set the snaplen to just catch and decode the ip layer info if you don't need the ports. you can also set the '-l' option to have it unbuffer it's output (it may be buffering tons of lines and sending them to you in one big chunk making the whole pipe and swapping issue worse).

If you just want to know that, then I'm sorry I don't know. :)

If you consider the TMTOWTDI, here is a couple of thoughs for you:

1. If you have a REALLY loaded network, write your own (or use one of the existing) daemons in C to store the data into the database (like MySQL) for example. You can later use Perl to extract the data, generate reports, and/or do any other fancy thingies.

2. Since you've mentioned that you are using a Linux box, you can use iptables(8) to do the counting. Check the manual for the --uid-owner and --gid-owner parameters. In this way, you will not have to write any C. You will also have very exact data and you will be use your hardware with most efficiency, since it'll be the kernel itself doing your job. :)

Leonid Mamtchenkov

Actually, I've looked into this. I've found a few solutions that modify the kernel, but they're really, really old. If I can't get this working, I'll probably end up using a netfilter patch that allows me to match the owner on incoming packets as well (owner-socketinfo, I think). But, don't iptables counts reset after a while?

Depends on what do you mean by "after a while". If that's service iptables restart, then, yes, of course. :)

Leonid Mamtchenkov