in reply to Extract new lines from log file
First, using the file size, as recommended above probably just solves your problem, so I'm just muttering academically here...
It sounds like you've got the remote files mounted on a network drive of some kind, right? Depending on the file system and weather or not it caches, your script might actually be downloading the entire file every time it reads it.
It might be easier to copy the file to a local partition and then process it.
-Pileofrogs
|
|---|