Re: Creating a Deamonized Log Scraper

Uh-oh, my chastened alarm is starting to buzz loudly!!! LOL. I think the both of you have some very good points and maybe that I haven't thought it through all the way. I did try to make this program so it would be non-system intensive...

My original instructions were to create a deamon that would continually scan the most current last line of seven database log files. Said databases are accessed continously and rather heavily. The entries to the logs are made as they occur, not really on a timed basis. I basicly want to proccess each new last line as it is generated and scrape only that new line for possible error codes. One of the requirements is that logdbd not miss any new last lines. By putting the scrape into an endless loop like that I thought I would be getting as close to real time error reporting as the cpu scheduler would allow, without having to scrape through the entire file to find the error on the last line. The sleep at the end is to give the operator some time to resolve the problem without flooding the system with emails, otherwise this daemon would keep sending messages every 100 microseconds until the message changed ( talk about resources!!! ). The regular expressions are there because the bulk of logfile entries are uninteresting and it's easier (and faster) to eliminate what I don't want to see then to look for the many errors I do find "interesting".

My manager and I had discussed the two ways to do this, my way and his, which was to create 1 single threaded proccess that sequentually checked last line on every database. I felt that I would not be meeting the instant reporting always "on" requirement that was being asked of us to deliver (what mesages might I be missing in log-1 while I cycled through log-50 thru log-100?. I really just want to get this thing working on ten logfiles so I can benchmark it and then write it his way and benchmark that.

I thank you for your input and am eager for any suggestions you might make in regards to this thought proccess. I am still new to this language and may be trying to misapply it in some manner here. Please let me know.
Thanks,
Pat

Comment on Re: Creating a Deamonized Log Scraper

Replies are listed 'Best First'.
Re: Re: Creating a Deamonized Log Scraper by ehdonhon (Curate) on Feb 08, 2002 at 17:36 UTC
But your method doesn't eliminate the possibility of missing lines. You still have a race condition. What if your log got 20 lines apppended to it during that 60 second sleep? You'd only see the last line. It sounds to me like you want to try to emulate 'tail -f' on your log files and then continuously cycle through them (perhaps in a multi-threaded fashion) looking for updates. I agree with the previous poster, though that a one to one relationship between processes and log files isn't such a good idea. I would suggest building a system where the number of children is configurable, and they all share the job of scanning your logs. Each time a child comes to a log it would need to pick up on the filehandle where it left off and scan till it gets to the end, then move on to whatever log it should do next.	[reply]

Replies are listed 'Best First'.

Re: Re: Creating a Deamonized Log Scraper
by ehdonhon (Curate) on Feb 08, 2002 at 17:36 UTC

But your method doesn't eliminate the possibility of missing lines. You still have a race condition. What if your log got 20 lines apppended to it during that 60 second sleep? You'd only see the last line.

It sounds to me like you want to try to emulate 'tail -f' on your log files and then continuously cycle through them (perhaps in a multi-threaded fashion) looking for updates. I agree with the previous poster, though that a one to one relationship between processes and log files isn't such a good idea. I would suggest building a system where the number of children is configurable, and they all share the job of scanning your logs. Each time a child comes to a log it would need to pick up on the filehandle where it left off and scan till it gets to the end, then move on to whatever log it should do next.

[reply]