in reply to Sharing data structures in mod_perl

The anecdotes I've heard are that Storable gets more and more expensive as you have to share bulkier data structures across. And that shared memory really bites, while disk storage is much preferred.

If your structures are complex enough, I've heard it was actually better to use a real database to share things! That is, for some amazingly small number of elements in a hash, it was faster to do a data query to PostgreSQL to fetch the interesting parts through a DBI query than to simply use Storable, write it to disk, read it from disk, and then look at it in Perl. Interesting scaling point. That stuck in my brain because it was counterintuitive at the time, both to the observer, and to me.

-- Randal L. Schwartz, Perl hacker

  • Comment on •Re: Sharing data structures in mod_perl

Replies are listed 'Best First'.
Re: •Re: Sharing data structures in mod_perl
by Hero Zzyzzx (Curate) on Mar 28, 2002 at 16:23 UTC

    That's why I went to this whole serializing system with Storable in the first place- I figured that saving DBI queries would always be a good thing, that is apparently NOT the case when you get large with your structures. I have a database (MySQL) that acts as my data repository- I figured I'd use data structures serialized on-disk to save time, avoiding DBI hits. This is all working, just not at the performance level I expected.

    It'd be interesting to find out where the break even point is- How much does serializing performance degrade as your structures get larger and larger? Does it degrade linearally or exponentially?

    I'm going to move back to DBI queries, or perhaps to a system that splits up my serialized structures into MUCH smaller chunks.

    -Any sufficiently advanced technology is
    indistinguishable from doubletalk.

[reply]
Re: •Re: Sharing data structures in mod_perl
by andye (Curate) on Mar 28, 2002 at 16:28 UTC
    Interesting scaling point.

    That it is. Presumably prepare_cached() and pooled connection handles (Apache::DBI) come into their own here?

    andy.

[reply]

      I've got Apache::DBI working, and I suppose I need to investigate prepare_cached() further. If you're familiar with prepare_cached, do you know if there are easy ways to deal with expiring stale statements, or do you have to roll-your-own staleness checker? A quick perusal of the docs didn't turn up a whole lot. . .

      -Any sufficiently advanced technology is
      indistinguishable from doubletalk.

[reply]
        You can use prepare_cached() without fear now. It used to be necessary to call finish() so that you wouldn't get a fatal error if one of your statements was called again before being fully read, but now it's just a warning. Make sure you have the latest DBI and you'll be fine.

        Also, use bind_cols() for maximum performance.

[reply]

In Section Seekers of Perl Wisdom