in reply to Downloading URL's in Parallel with Perl
Remember your Von Neumann bottleneck, it is doubtful that what is slowing down your task is the overhead of processing the data, it is much more likely that the bottleneck exists in the actual data pipe (in other words, not processing BUT bandwidth!) And attempting to stuff 10k/sec of data down a 1k/sec pipe won't make the pipe bigger... it may actually end up slowing down your overall wall-clock time due to TCP collisions and other assorted baddies. I'm a big fan of parallelization throughout... just make sure it makes *sense* in your particular configuration.
|
|---|