|No such thing as a small change
Re: Extract new lines from log fileby pileofrogs (Priest)
|on Jan 02, 2007 at 23:17 UTC
First, using the file size, as recommended above probably just solves your problem, so I'm just muttering academically here...
It sounds like you've got the remote files mounted on a network drive of some kind, right? Depending on the file system and weather or not it caches, your script might actually be downloading the entire file every time it reads it.
It might be easier to copy the file to a local partition and then process it.