in reply to Get specific entries from a file which is always growing

Assuming this is linux (use cygwin otherwise), shell commands are made for this: 
  grep "IDENTIFIER" File1 > File2
  tail -f File1 | grep "IDENTIFIER" >> File2

[download]

            "XML is like violence: if it doesn't solve your problem, use more."

Replies are listed 'Best First'.
Re^2: Get specific entries from a file which is always growing
by ashok.g (Beadle) on Jun 10, 2011 at 17:49 UTC
    This doesn't seem to be working. I am not getting any data in the File2 using these commands.
[reply]

      I suppose you're talking about the second command. The first grep command should definitely put something in File2, as long as there's some occurrences of "IDENTIFIER" in File1 at the time you run the command.

      The "problem" with the second command line is that when redirecting grep's output to a file, the stream will be fully buffered. This means that nothing will be written to File2 before the buffer (typically 4K in size) is full, or tail closes its pipe to grep (the latter won't happen when tail is running in "follow" (-f) mode).   Depending on how much output grep produces with your data, filling the buffer may take a while...

      Whether the buffering is a problem depends on your expectations. If you want File2 to be updated immediately with every newly found occurrence, then the above shell command is not the right tool for the purpose. In this case, you could write the tail/grep combo yourself in Perl (as already suggested), in which case you do have control over buffering (i.e. you can flush data immediately).

[reply]
[d/l]
[select]
        I haven't actually tried this, but I am told that "tail -f" calls fflush by default, so it avoids the buffering issue you describe.

        grep's "--line_buffered" option is off by default, so these commands should not be "suffering from buffering".

        I'm hoping some gurus with a deeper understanding of these issues will present the canonical way to resolve this, hopefully with a succinct explanation. I did try to RTFM on the Internet, but I guess my attention-span was insufficient to grok.

        There is also an "unbuffer" command that could potentially help this situation. I Will try to RTFM at a more civilized hour.

                    "XML is like violence: if it doesn't solve your problem, use more."

[reply]

In Section Seekers of Perl Wisdom