Re: Very fast reads from an external program

Well, if you are going to write the output to disk file, the program will even be slower, as disks are orders of a magnitude slower than memory. If you are writing to memory, you quickly have a problem because the large amounts of data tcpdump is writing will quickly fill up large chuncks of memory. If it's indeed the buffer size that is the problem, you're better off enlarging the buffer size, by tweaking the OS.

But I expect that this is one of the cases one would prefer to use C instead of Perl.

Abigail

Comment on Re: Very fast reads from an external program

Replies are listed 'Best First'.
Re: Re: Very fast reads from an external program by MarkM (Curate) on Apr 26, 2003 at 03:45 UTC
You should note that I suggested the use of a file system that performed deferred writes, or that are based from virtual memory, and not a real disk. In any case, the pages just written are still likely to be in RAM, meaning that the write ahead is to RAM, and the read behind is from RAM. The real benefit of using a true file, is that the system can write the data to the file at its leisure, marking pages as clean and ready for reclaimation as necessary. Using a shared memory segment, for example, requires that the processes perform their own scheduling to ensure that the producer does not over-write pages not yet read by the consumer (during a fast network burst, for example). Using a file lets the kernel do this magic for us.	[reply]

Replies are listed 'Best First'.

Re: Re: Very fast reads from an external program
by MarkM (Curate) on Apr 26, 2003 at 03:45 UTC

You should note that I suggested the use of a file system that performed deferred writes, or that are based from virtual memory, and not a real disk. In any case, the pages just written are still likely to be in RAM, meaning that the write ahead is to RAM, and the read behind is from RAM. The real benefit of using a true file, is that the system can write the data to the file at its leisure, marking pages as clean and ready for reclaimation as necessary. Using a shared memory segment, for example, requires that the processes perform their own scheduling to ensure that the producer does not over-write pages not yet read by the consumer (during a fast network burst, for example). Using a file lets the kernel do this magic for us.

[reply]