in reply to buffering from a large file
But it seems you're only interested in every 4th line, and while it's fairly easy to find lines when starting to read from the middle, you cannot know which line will be a 4th one without having counted from the beginning.
So, perhaps you should first look where the bottleneck is. CPU? IO? If it's CPU bound, using threads or forks may improve things. If it's IO bound, then using multiple threads/processes using different controllers may help. But only if they read different parts of the file, and then you're back to the sync problem. And then there are still many factors that play a role (one disk, multiple disks, mirrors, striping, disk/controller caches, other IO, etc) determining how much there's to gain.
|
|---|