Trying to identify unique lines in a log file

aditya1977 has asked for the wisdom of the Perl Monks concerning the following question:

I'm writing a program that will parse a log file and write data to a MySQL database based on the content of the log file.

The script will run via cron and I want to make sure I skip lines that have been parsed before.

One idea I had was to base 64 encode the line and make this a unique key in the database. Then the next time the script runs, if there is a matching key, skip the line.

What I don't like about this is that the encoded line is quite a long string, which would make viewing the database difficult to read. Is there a way to encode a long string to a relatively short string string?

Or perhaps there is a better way to do this?

Comment on Trying to identify unique lines in a log file

Replies are listed 'Best First'.
Re: Trying to identify unique lines in a log file by choroba (Cardinal) on Mar 05, 2015 at 10:39 UTC
For shorter strings, see Digest::MD5 and similar. Also, if the log grows on one end only, it should be enough to just remember the last line. لսႽ† ᥲᥒ⚪⟊Ⴙᘓᖇ Ꮅᘓᖇ⎱ Ⴙᥲ𝇋ƙᘓᖇ	[reply]
Re^2: Trying to identify unique lines in a log file by aditya1977 (Novice) on Mar 05, 2015 at 10:57 UTC
Thanks! You made me realise that there's no reason not to use hashing rather than encoding. And MD5 hashes are significantly shorter. Timestamp entries are present, but not unique as some events take place milliseconds apart.	[reply]
Re: Trying to identify unique lines in a log file by Theodore (Hermit) on Mar 05, 2015 at 10:37 UTC
Is there any timestamp in the log? If not, can you add one? Or even, Why not cycle the log first (copy and truncate) and then parse the "static" recycled log?	[reply]
Re: Trying to identify unique lines in a log file by GotToBTru (Prior) on Mar 05, 2015 at 19:45 UTC
To reprocess the log file each time seems a terrible waste of time and resources, not to mention storing your ever-growing index of lines you have already logged! Use `seek()` and `tell()`. You need only store a bookmark and you will know where to start from next time. This will need to be coordinated with whatever process truncates your log file, of course. Dum Spiro Spero	[reply] [d/l] [select]