princepawn has asked for the wisdom of the Perl Monks concerning the following question:

In a large system, you want all of your database queries sequestered to some sort of cached , labelled repository, accessible by name and perhaps parameterized for a few placeholder fields: 
my $records = pull_query 'wool-sweater-orders', within => 'last 5 days
+';  # either finds and runs query or retrieves cached, self-expiring 
+result
while (my $record = $records->next) {
   warn $record->{brand}, $record->{price};
}

$records = 'all_employees', age => 26

[download]
I am aware of Class:Phrasebook::SQL, but was wondering what methods you guys found useful for cataloging and storing database queries.... note the emphasis in on SELECTs as most production systems will do all database updates via PL/SQL or via some sort of batch insert at regular intervals as inserts are resource-intensive.

Replies are listed 'Best First'.
Re: how do you label and cache your database queries?
by perrin (Chancellor) on Sep 01, 2001 at 22:34 UTC
    I don't agree about most production systems using stored procedures or timed jobs for all updates. I've never seen a system that did that, although I've often heard DBAs wish that things were done with PL/SQL. I think it's better to avoid stored procedures as much as possible, but that's a whole different topic.

    In my systems, I write Perl classes for each of the data objects I'll be manipulating. These classes handle their own caching.

    For example, if I'm building a store and I manipulate data for products, I would make a Product class with load() and save() methods. I'd pass an ID to load. The load() method can then check the cache and go to the database if the object I want is not cached. If it does go to the database, it puts the retrieved data in the cache for future use. I would use the object ID and some constant string denoting the object type as my key in the cache.

    It would be very easy to build this kind of thing using SPOPS, and some other tools like Alzabo already do it. My homegrown approach used SQL statements coded in the classes themselves rather than trying to auto-generate them.

[reply]
      I'm very green at OO perl, so I can't address the main topic . I would like to provide some perspective from the DBA side.

      In an OLTP database, stored procedures, and triggers have to be very carefully done, or they can cause more performance problems than they solve. SP's and triggers can also cause a lot of headaches when the database is being replicated.

      As DBA,I ask my perl folks to optimize their SQL and tell me the end requirement for delivery of the info. I'll opt tune indexes if needed and load balance large batch jobs to minimize impact on response time.

      We now return you to princepawn's original topic...

      Dave

[reply]
Re: how do you label and cache your database queries?
by princepawn (Parson) on Sep 01, 2001 at 23:01 UTC
    I don't agree about most production systems using stored procedures or timed jobs for all updates. I've never seen a system that did that, although I've often heard DBAs wish that things were done with PL/SQL.
    Actually, that is our group policy. Selects are done in Perl and anything which alters the database is done via a large library of on-disk stored procedures. Having them on-disk, keep the "business objects" you discuss below out of our already-cramped main memory.

    Regarding timed jobs, I once worked at a vote-polling company and we simply needed to push an entirely static website every hour or so. So, we didn't need event-by-event database updates. Just queue them all in a file and do a SQL_LOAD. Also, any digital music company which receives a feed of new music products would more than likely convert this to a SQL loader file and then use the built-in tools. Broadvision, a C++ app server, for instance has this bv_load program that loads its tables much faster than you could do with Perl.

    I think it's better to avoid stored procedures as much as possible, but that's a whole different topic.
    I hate PL/SQL. I hate writing it. But, I have to admit that server-side processing is much faster than back and forth between client and server.
    In my systems, I write Perl classes for each of the data objects I'\ ll be manipulating. These classes handle their own caching.
    Let's call these "business objects". This is a good idea. It sounds like the same thing that the BingoX framework is touting, but reportedly their actual ease-of-use is quite low.
    denoting the object type as my key in the cache.
    And I assume you use Cache::Cache to store your data.
[reply]
[d/l]
      I actually wasn't using Cache::Cache last time I did this, although that would be a good choice. I was using BerkeleyDB.pm, which allows you to specify an amount of memory to use as a shared buffer and keeps the rest on disk. Cache::Cache or MLDBM::Sync would actually be much easier and may perform just as well.
[reply]
Re: how do you label and cache your database queries?
by derby (Abbot) on Sep 02, 2001 at 04:09 UTC
    PP

    Is cacheing the result set really what you want? I know on web-based systems, I'd prefer to cache at the page level (or even a component level).

    If it's an oracle database (PL/SQL), I think I'd let Oracle cache through decent SGA settings (caveat - I'm not a dba). Is the network traffic really a bottle neck anymore?

    -derby

[reply]

Back to Seekers of Perl Wisdom