Better Way of Storing Data in a Text File?

jupe has asked for the wisdom of the Perl Monks concerning the following question:

I am currently looking at revamping a project I did about 2 years ago. At that time I put together a website that lets people enter information about what they are doing on any given day (this was a corporate project). When the Perl program assembled the record, it looked something like, "date,user,comments", with each record on a separate line. When I wanted to parse the information, I had the Perl script open up the text file that these records were stored in, shove them into memory as a hash, and pick out which ones I wanted.

It was simple, quick and easy to develop. Times have changed (for me) and I am looking for a more efficient/elegant solution, though. There are a couple of constraints: 1. No database, and 2. I cannot gaurantee the order of the records at this time.

As it stands, I sort the records by the date in memory. A user has the ability to change dates on existing records or enter records for past/future times. This means that I never know where a record in this text file might be, since a record may be appended to the file with a date drastically different then the surrounding records. So I use a linear search routine and sort after the fact.

For some reason this just offends my sensibilities.

The project is not slow (even after two years worth of data), it is not outdated, and is very useful. But I am still bothered by the way I am storing the data and how I am searching through it. There must be a better way! So my question really is this:

What is the most efficient/elegant way of storing data in a text file so it can be easily and quickly queried without setting up a full-fledged database?

Thank you all for your help!

Comment on Better Way of Storing Data in a Text File?

Replies are listed 'Best First'.
Re: Better Way of Storing Data in a Text File? by Cody Pendant (Prior) on Dec 31, 2002 at 00:55 UTC
I don't know about the actual storage, but you can abstract yourself from the actual data by using modules like Tie::File -- or you can use a module which allows you to treat a text file as an SQL database, but I get the feeling that's not thing that's bothering you. It would still be a big ugly text file, but you wouldn't have to think about it as much. Update: DBD::File and DBD::CSV would be the modules to check out for that text-as-SQL thing. When you say no databases, do you mean not even dbm-type databases? -- `($_='jjjuuusssttt annootthheer pppeeerrrlll haaaccckkeer')=~y/a-z//s;print;` [download]	[reply] [d/l]
Re: Better Way of Storing Data in a Text File? by Three (Pilgrim) on Dec 31, 2002 at 14:11 UTC
I like to use DB_File for simple database use. It uses berkley db and ties it to a hash. Here is a quick example. `#Get package by ppm install db_file use DB_File; #Decalare varable my %hash; #Tie db to hash tie(%hash, "DB_File", "my.db"); #Add record $hash{'1'} = "hello"; #Check for existance if(exists($hash{'1'})) { #Remove record delete $hash{'1'}; } #Untie from db untie %hash;` [download] Hope this helps.	[reply] [d/l]