in reply to perl performance vs egrep
I belive that the bottleneck in such a search is probably disk access. When I grep through between 10M and 100M of data, it seems that it's always much faster if I rerun the search, as the files are cached in memory at that time. Egrep is very fast, as it builds a dfa, so it's probably faster than perl; however, as you probably can't hold all 20G of data in the memory, the difference will be probably negligable.
Update: as a consequence, if you want to make grepping fast, you'll have to make disk access faster; make sure dma is set up correctly, put the 10 files on different disks if you can, copy the data to hard disk if they are on cd or dvd or nfs now. Also, do multiple searces at once if you can. (You're not zgrepping compressed files, are you? That would slow things down.)
Update 2: as for your update: to tell how much memory they are using, use top or ps if this is on a unix box.
|
|---|