Tailing/Streaming a Log File into Any Web Browser from Daemon

hackdaddy has asked for the wisdom of the Perl Monks concerning the following question:

I have a daemon that runs jobs. While the job runs, it generates a log file. I would like to stream or tail that log file to users' web browsers.

I have experimented with the following recipe, but it does not show the entire log file:

POE Cookbook Tail Following Web Server

Ideally, I would like to create an HTML table and send chunks of the table to the browser as "tail -f mylogfile.log |" was open. I attempted this with a small hack of the HTTP::Daemon sample code. However, with Microsoft IE the table is not shown in the browser until tail stops sending HTML data. FireFox, surprising, does exactly what is expected and creates the table on the fly, but stops at approximately 16561 lines in the table.

Sending plain text seems to work, but it would be of greater utility to users to parse the log file and color code particular entries, etc, using HTML.

I've solved this particular problem before by sending chunks of JavaScript to the browser to append text in a textbox. I suppose I could do the same thing with AppendChild in DOM, but I dislike that solution. There is also the refresh option, but that is an expensive operation if your log file will be tens of thousands of lines long.

Tailing a file through a browser is a great utility. There has to be an easier way to to this. I want to be able to accomplish this in any popular browser.

Any assistance or suggestions is greatly appreciated.

Thanks.

P.S. As an after thought, is there any way to do this with AJAX?

Comment on Tailing/Streaming a Log File into Any Web Browser from Daemon

Replies are listed 'Best First'.
Re: Tailing/Streaming a Log File into Any Web Browser from Daemon by blm (Hermit) on Nov 17, 2005 at 07:57 UTC
I am interested in doing a similar thing at the moment but it is only a spare time interest so I havent progressed far. It was all brought on by this post on livejournal and some rather restrictive IT policies at work (No ssh). It mentions doing a tail log thing using cgi and AJAX but the code is not downloadable.	[reply]
Re: Tailing/Streaming a Log File into Any Web Browser from Daemon by strat (Canon) on Nov 17, 2005 at 11:15 UTC
Ajax also came into my mind. I think you can do it. (Do you know CGI::Ajax? But I have never tried it in combination with POE.) Best regards, perl -e "s>>F>e=>y)\martinF)stronat)=>print,print v8.8.8.32.11.32"	[reply]
Re: Tailing/Streaming a Log File into Any Web Browser from Daemon by BUU (Prior) on Nov 17, 2005 at 08:33 UTC
Web browsers will typically not render tables until the entire code has been read. This is generally considered to be a feature. Your best result would probably be to send plain text, but the javascript should also work fairly easily. Another solution might be to only display the last 15 or so lines and have it automatically refresh, this way you can display colorized logs without waiting for the entire file to display.	[reply]
Re: Tailing/Streaming a Log File into Any Web Browser from Daemon by neosamuri (Friar) on Nov 17, 2005 at 07:08 UTC
The first thing that comes to my mind is that instead of using the -f flag, just have the page refresh itself with the reload http header of the meta html header tag. meta tag	[reply]
Re: Tailing/Streaming a Log File into Any Web Browser from Daemon by ickyb0d (Monk) on Nov 17, 2005 at 15:52 UTC
well here's what i'm thinking to solve this. when the initial page is loaded, create a file called log.old containing the current contents of the log file. Then have an ajax function run on a javascript refresh (every 5,10,30 seconds or whatever). If you anticipate a lot of date being transferred (2000+ chars or so) be sure to make the ajax function a post function, rather than get. have this ajax function call a perl script that populates a new version of the log file, log.net. then run a 'diff log.old log.new' and store the result in a file/variable' then move log.new to log.old. doign this will help cut down on transferring the whole file every time. then just return the diff text from the ajax function using the returned text, simply append it to either a div or a text area on that webpage using various DOM techniques I'm fairly confident that this would work, it's just a matter of putting all the right pieces together. for all the ajax functions check out XMLHttpRequest	[reply]