I'm failing to see the compelling reason for designing your system in this fashion. I would look at doing something that has the following characteristics:
  1. Is encapsulated. You (the requesting script) do not know how the thing does what it does. All you care about is that it does what it promises to do, which is retrieve your data.
  2. Is fast. You want it to give you the data you request in a minimum amount of time.
  3. Is fast. You want it to load in a minimum amount of time.
  4. Is small. You want it to use the least amount of memory.
Sounds pretty tough, huh? Well, it's not. What you are looking for is not a datastructure, but an object.

YAY-US! You, too, can be a part of the O-O revolution, my friend! You can be HEE-ULLED of your pro-see-ju-rull ways!

What you're looking for is not something that loads all your data at once. That is waaay too slow to load, as I'm sure you've noticed already. You're looking for something that will cache data.

Now, others have suggested using DBI's caching, and that's good, or some sort of memorize, and that's good, too. I'm suggesting a third method, and that is to write an object that will hide your data-retrieval methods from yourself.

The basic concept is this - you instantiate this object. Then, when you need some data, you ask it for that data, and only that data. It will then check to see if it has it. If it doesn't, then it will go out to the database, get the data, store it within itself, then give it to you. Now, if you ask for that data again (for whatever reason), you will get the data immediately. You don't store the data ... this object does.

This method immediately allows for three things:

  1. You get rid of all those nasty globals. Now, all you have is a file-scoped lexical (the object) that will handle all your data needs.
  2. You can request the same data over and over and not incur a performance penalty. This means that your logic flow is cleaner and clearer. Your routines are more loosely coupled. (This is a good thing, in case you're wondering.)
  3. If you have more than one script that uses these data structures and, because you will, you end up changing them, you only change stuff in one place! Think about that - maintenance is made 10x easier. I know I always like that.

Now, you're gonna say "Well, I wrote the object, so I'm storing the data. You're just making my life more complicated."

My answer is simple - "No. You are the script that needs the data, or the general. The object is someone else, a quartermaster if you like. Even though the general puts the quartermaster in his position, he still has to requisition supplies through a known and agreed-upon method."

------
/me wants to be the brightest bulb in the chandelier!

Vote paco for President!


In reply to Why are you even bothering to do it that way?!? by dragonchild
in thread Using tie to initialize large datastructures by htoug

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.