Assuming other applications visiting this logfile are playing nice too, you could use flock with the LOCK_NB bitflag set (that is Lock Non-Blocking). This will cause flock to return immediately instead of waiting for a lock. When it returns, its status will tell you whether it was granted a lock or not. If not, you know something else has the file locked. Flocking is voluntary, so scripts/programs that don't lock the file or check to see if it's locked won't play nice with applications that do.

As far as reading lines as they're being written, that's precarious. Due to buffering, you're unlikely to get line by line info, but rather chunks every so often.

By the way, another option would be to periodically check last modification time, or file size. A change there will indicate the file has been written to.

Perhaps the easiest way would be to keep a record of how much of each file you have already read. Then your test for it having been written to, is just checking the current size of the file. A tied hash might be one way of persisting this information.

Perhaps the harder thing is to determine when you've read all you are going to get. Ie. Distinguishing between your reader having caught up with the writer and the writer having finished.

If you are reading the files in seperate threads, then you can afford to have the thread hang around waiting for the file to grow some minimum amount before reading the next line, and if it fails to meet that minimum within a preset amount of time, you read and process whatever is left and call it quits.