Active Munging via a pipe (code)

deprecated has asked for the wisdom of the Perl Monks concerning the following question:

I run two services which produce log files on the scale of multigigabytes: Postgres and http://opennap.sourceforge.net/opennap. Taking huge zipped up tarballs of these logs and parsing them during off hours or when I come home from work gets tedious. I'd rather just have a perl daemon set up that actually read them as they were generated and stuffed em in a database in the format I want.

So the approach I came up with was using mkfifo(1) and this eensy-weensy loop:

#!/usr/local/bin/perl

open FIFO, "<foo" or die "$!\n";
while (<FIFO>) {
  chomp;
  print "$_\n";
}
[download]

The problem is, this doesnt actually stay open and alive. It dies after the first "echo 'hello' > foo".

Is there a way I can make perl "listen" to a file and actually have it be parsing data and doing inserts on a db?

thanks,
brother dep.

--
Laziness, Impatience, Hubris, and Generosity.

Comment on Active Munging via a pipe (code) Select or Download Code

Replies are listed 'Best First'.
(Ovid) Re: Active Munging via a pipe (code) by Ovid (Cardinal) on Jan 17, 2002 at 00:03 UTC
This type of stuff isn't exactly my strong suit, but it sounds like the the while loop is exiting on `EOF`. If that's the case, you need to clear the `EOF` flag. The classic example, modified from the Cookbook: `for (;;) { print while (<FIFO>) sleep FOR_A_WHILE; # some constant seek <FIFO>, 0, 1; # resets EOF }` [download] Is that what you are needing? It should effectively be a `tail`. Cheers, Ovid Join the Perlmonks Setiathome Group or just click on the the link and check out our stats.	[reply] [d/l]
Re: Active Munging via a pipe (code) by PrakashK (Pilgrim) on Jan 17, 2002 at 00:15 UTC
How about using File::Tail? `use File::Tail; $file=File::Tail->new("/some/log/file"); while (defined($line=$file->read)) { print "$line"; }` [download] HTH, /prakash	[reply] [d/l]
Re: Active Munging via a pipe (code) by broquaint (Abbot) on Jan 16, 2002 at 23:45 UTC
The problem with looping on the input from named pipes is that after the first read you're left with an `EOF` for the second read¹, so the loop exits. Subsequently you can't loop on a named pipe without working some serious voodoo (over-riding `CORE::GLOBAL::eof()`?). So this provides with many excuses to do dubious things such as using `goto()`, labels and (dum dum dum) infinite loops. Personally I'd go with an infinite loop opening the FIFO on each iteration, or take have a good long think about it. Good luck! HTH broquaint ¹unless you have something writing faster than perl can read	[reply]
Re: Active Munging via a pipe (code) by Zaxo (Archbishop) on Jan 17, 2002 at 06:51 UTC
Here's yet another way, demonstrated from the shell: Make the pipe, and... `$ mkfifo logpipe $ perl -e 'open PIPE, "< logpipe"; { select 1<<fileno(PIPE),undef,unde +f,undef; print <PIPE>;redo}' &` [download] ... start a simple server in the background. `select` makes it sleep till `PIPE` is readable. `redo` in a bare block loops forever. Kick the tires: `$ echo foo >logpipe foo $ echo far >logpipe $ far echo fie >logpipe fie $` [download] The asynchronous operation of the server is shown in the staggered output. Clean up with $ fg perl -e 'open PIPE, "< logpipe"; {select 1<<fileno PIPE,undef,undef,un +def;print <PIPE>;redo}' # do a Ctrl-C here $ [download] I think this is the Right Way® to handle your log file problem. Sleeping till there is something to read will save a lot of load on the host. Update: Changed to use mkfifo instead of mknod, and repaired a pasto in the timeout arg of select. After Compline, Zaxo	[reply] [d/l] [select]
Re: Active Munging via a pipe (code) by count0 (Friar) on Jan 17, 2002 at 01:54 UTC
Since nobody has mentioned it yet.. it's definitely worth noting that perlipc has a section on named pipes (FIFOs).	[reply]
Re: Active Munging via a pipe (code) by gt8073a (Hermit) on Jan 17, 2002 at 00:18 UTC
Is there a way I can make perl "listen" to a file and actually have it be parsing data and doing inserts on a db? This is seriously off the top of my head, and I have never done anything similiar, but instead of using open( BLAH, $file ) and while ( <FIFO> ) couldn't you use sysopen, sysread and then sleep on eof? You can "know" where you are in the file, and just sleep when you get an EOF. I am sure there are buffering and pre-munging munging issues involved, but maybe this will help. Sorry, I do not have any code to help you with, but someone else prolly can . . . Will perl for money JJ Knitis (901) 756-7693 gt8073a@industrialmusic.com	[reply]