Yoda has asked for the wisdom of the Perl Monks concerning the following question:

I was looking for a flatfile database module (something that would be platform independent and database independent). I searched through CPAN and didn't find what I was looking for. I was thinking of doing something like this:
Forgive me if this doesn't run as typed. At the moment, I have nothing to test it on.

open DBFILE "< file.dat";
my $cnt = 0;

while (defined(<DBFILE>)) {
  chomp;
  ($rec[$cnt]{'Name'}, $rec[$cnt]{'Age'}, $rec[$cnt]{'Address'},
  $rec[$cnt++]{'Phone'}) = split(0036,$_);# Octal 36 is ASCII RS I am using as a record separator
}
close DBFILE;

That is how I would place my data in memory. The array of hashes seemed to work as I was playing around. My questions are:

If there were say 1000 records, would this array of hashes be a memory hog?

Am I doing this the hard way? I have a talent for that.

Is there a good Module out there for flat file databases?

Yoda

Replies are listed 'Best First'.
Re: Flat File Database
by kschwab (Vicar) on Jan 31, 2001 at 04:11 UTC
    If your needs are simple, you may like SDBM_File. It's cross-platform, as you mentioned, but is limited to retrieval via one key only. It uses the tie interface, so you wouldn't need to deal with the I/O guts. Just read and write hash keys.

    If your needs are more complex, some other ideas:

    • The FreezeThaw Module (keep your data in perl data structures, and freeze/thaw them in a form suitable for file i/o)
    • Storable - similar to FreezeThaw, perhaps faster, file i/o built-in to the module.
    • DBD::CSV Use SQL against CSV (comma separated value) files
[reply]
Re: Flat File Database
by mirod (Canon) on Jan 31, 2001 at 04:02 UTC

    You could use DBD::RAM (soon to be superseded by DBD::AnyData), which let you use CVS, XML and a number of other formats through the DBI interface.

[reply]
Re: Flat File Database
by Trinary (Pilgrim) on Jan 31, 2001 at 04:17 UTC
    DBD::CSV would probably be a good place to start for handling this stuff in a sane fashion.

    Re: memory usage, it'd be far easier to suggest a better approach if we knew how you were going to be manipulating the data. For example, if you're just iterating through, then you can use the DBI functions to get an array of each row, etc. No hoggin mem there.

    Ah yes, formatting...there's a handy pair of tags you can put into your posts: <code> and </code>

    Works quite well, you don't have to worry about escaping HTML or any of that stuff inside code tags.

    Trinary

[reply]
      Thanks everyone for the info. I will get the hang of this soon. The CODE tags, I wish I would have known sooner. It took me about 30 minutes to format my script. Thanks to all! DBD::RAM or DBD::CSV looks like what I need, but I will look at all of the suggestions.
[reply]
Re: Flat File Database
by flocto (Pilgrim) on Jan 31, 2001 at 04:13 UTC
    You should think about using DBM. The most common flatfile-DB is the "DB" (Berkeley DB library). The necessary modules are installed on most systems. Just try the following:
    use DB_File;
dbmopen %hash, $file, 0666 or die "This is my very own error-message: 
+$!\n";

$value = $hash{'key'}; # get data
$hash{'key'} = $value; # put data
delete $hash{'key'};   # delete data

dbmclose %hash;

    [download]

    There are a lot different flat-file Databases and you might need another one for your needs. More Information can be found online at CPAN or in O'Reilly's "The Perl Cookbook", Chapter 14 (page 489)
    Regards, octopus
    --
    GED/CC d-- s:- a--- C++(+++) UL+++ P++++$ L++>++++ E--- W+++@ N o? K? w-- O- M-(+) V? !PS !PE !Y PGP+(++) t-- 5 X+ R+(+++) tv+(++) b++@ DI+() D+ G++ e->+++ h!++ r+(++) y+
[reply]
[d/l]
Re: Flat File Database
by Anonymous Monk on Jan 31, 2001 at 09:25 UTC
    Check out this one. Fully documented and it works great on any platform. http://www.webreview.com/1998/10_09/developers/10_09_98_2.shtml
[reply]
Re: Flat File Database
by Steeeeeve (Initiate) on Feb 02, 2001 at 02:36 UTC

    Hmmm, 1000 records. That is a rather small database. It is large on a Pc scale, but small on a business scale which most of the SQL drivers and technology are targetting.

    I once developed a module called Lady /TM to manage small persistent data indexes for a website. The use of the indexes came out of realizing that precomputed cache's can be time-saving. Actually they were crucial to the success of the project.

    With Lady /TM there is compatibility with EZDB (which you could also use.) EZDB is at EZPERL.com. It is a flatfile database manager that you can use online. This means you can set it up via FTP access in a remote CGI bin and use the WWW to manage your live data. Lady /TM has no interface like EZDB. EZDB requires that you setup templates for display and entry. With Lady /TM I write scripts that need fast access to site-related data in prototyping. Lots of site data is purged, keeping the datafiles small.

    Like a flatfile, Lady /TM keeps a header of the top line. Something like:

    pkey|index|value1*~*value2*~*value3*~*value4*~*value5*~*|blank|

    The table methods let you grab the string values from a hash after it is initialized. Processing the string is easily done with conventional flatfile processing methods.

    With enough interest, I will develop the needed code to allow creation of forms, designating the index and the order of the rest of the fields and values within the flatfile record.

    Lady /TM was posted to this forum under the Utilities scripts.

    -Steeeeeve

[reply]

Back to Seekers of Perl Wisdom