Re: Generic repository of files

You could possibly use the DBI module / system for a level of abstraction (and for even more abstraction, subclass Class::DBI. This may be helpful to you because of the myriad DBD drivers designed to work with DBI, which include:

DBD::Anydata: DBI access to XML, CSV and other formats.
DBD::SQLite: A single-file, self-contained RDBMS in a DBI driver.
DBD::MySQL (and other more heavy-weight databases)

My point is that if you start with the framework for using a database, you can keep it as generic as possible at the higher levels of abstraction, and with minimal changes, substitute any of the wide range of DBD database drivers.

If you wish to layer DBI over flat files, you can start with DBD::Anydata. When it comes time to graduate to a real database, you can use DBD::SQLite, and when that doesn't keep you afloat you can switch to MySQL, PostgreSQL, or some other real database.

Dave

Comment on Re: Generic repository of files

Replies are listed 'Best First'.
Re^2: Generic repository of files by Tanktalus (Canon) on Jan 26, 2005 at 00:04 UTC
Interesting. I can just see it now: DBD::FTP, DBD::HTTP, ... ;-) I'm not actually interested in parsing or dealing with the internals of these files. The files would simply be BLOBs in a database. Think of this more as systems management - the task at hand is simply to store one or more files together so that they can be retrieved together. These files could be RPMs or the like (more like tarballs in my case), but the code I'm looking at is not actually interested in the contents of those tarballs, just an easy way to copy them into multiple locations - think, for a second, of the CPAN archive as an example. We want to add a bunch of tarballs to the repository, and then later pull them out - you want 10, I want 15, and between us a few are the same. No point in both of us going to the originator of those few tarballs to get them, we'll all go to the repository to get them. This gets even more important the more people there are, imagine being the owner of File::Spec or FindBin or other such modules and being bugged by every perl developer to get your modules. The repository takes care of all that work. Our current process goes and recreates the tarball each time it's needed. I want to nix that because we can be creating the same tarball 20+ times per day. A repository would, of course, help here. We'd create each tarball once, put it in the repository, and then extract it from there each time it's needed.	[reply]
Re^3: Generic repository of files by stvn (Monsignor) on Jan 26, 2005 at 01:44 UTC
Interesting. I can just see it now: DBD::FTP, DBD::HTTP, ... ;-) That's actually not a bad idea. The trick is how to make FTP and HTTP interactions into a SQL like dialect. For inspiration, take a look at DBD::Google and the more recently released DBD::iPod, or the older, and kinda funky DBD::Chart. And actually DBD's are not really as hard to write as you might think. There is plenty of good documentation, and solid implementations out there which you can work from. Personally I always look at DBD::mysqlPP whenever I need a reference, the code is really clean and very easy to read. -stvn	[reply]