in reply to Best complex structure?

A couple of thoughts. Your top-level keynames appear to serve two purposes:

  1. ordering your data;

    But this requires you to sort the hash to recover the ordering.

  2. keeping a count of the number of values returned by your 3 queries;

    But it forces you to sort and count the values Which you've recognised by considering accumulating the counts within separate hash keys.

A better solution to both problems is to use arrays. Actually an Array of Arrays (of Arrays) of Hashes.

@structure = ( [ # Result from first query. { ... }, { ... }, ], [ Results from second query [ # first parameter { ... } { ... } ], [ # second parameter { ... }, { ... } ] ], [ # results from third query [ # first parameter { ... } ], ... ] );

You may be able to flatten the inner levels of the last two query results -- it's not clear from your post whether you need this separation.

This way, you neither need to sort to recover the ordering nor count to discover the numbers. The ordering is inherent and the counts are mantained by the individual arrays. The worst you might need to do is accumulate the total counts for a composite query:

# Total results from all parameters of query 2 # See List::Util qw[ sum ]; my $q2Total = sum @{ $structure[ 1 ] };

The second thought is that your using fetchrow_hashref in a tight loop and pushing to an array.

From the DBI pod:

If $slice is a hash reference, fetchall_arrayref uses fetchrow_hashref to fetch each row as a hash reference. If the $slice hash is empty then fetchrow_hashref() is simply called in a tight loop and the keys in the hashes have whatever name lettercase is returned by default from fetchrow_hashref. (See FetchHashKeyName attribute.) I

Using the correct varient and parameters of the fetchall_arrayref will probably run more quickly and avoid a lot of copying.

Lastly, without knowing what volumes or the nature of the data your pulling in, this maybe off target, but unless you absolutely need to have access to all this data simultaneously, and the processing that requires it is such that it cannot be done by the database itself, you probably ought to consider fetching that subset of the data you need at any given time rather than fetching it all in one big hit and storing it locally. Especially so if this is a (non-mod_perl) CGI program where your going to have to reload it for each request.


Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
"Science is about questioning the status quo. Questioning authority".
The "good enough" maybe good enough for the now, and perfection maybe unobtainable, but that should not preclude us from striving for perfection, when time, circumstance or desire allow.

Replies are listed 'Best First'.
Re^2: Best complex structure?
by mrborisguy (Hermit) on Jun 08, 2005 at 18:10 UTC

    I would definitely go with BrowserUK's idea. Since you are numbering anyway, an array is perfect for you. Basically, the "3.1" is only important because it comes after "2.*" and before "4.*", so there's really no reason to put it in a hash, but rather an array is good.

    BrowserUK: Does your example have one to many levels of anonomous arrays? I would think it would more be something like

    @structure = ( { ... }, [ { ... }, { ... }, ], [ { ... }, { ... }, ], [ { ... }, { ... }, ], );

    That way you can do something like:

    my @structure = ( $hashref_from_first ); foreach @query { ... push @structure, [] foreach @subquery { ... push @structure->[0], $hashref; } }

    Or something like that (but probably a lot better)

        -Bryan

      As I mentioned in the post, it wasn;t clear to me if the OP needed to retain the 2.1/2.2/2.3 etc. levels arising from the multiple executions of the second and third statements with different bound parameters. Hence I added an extra level to retain them and noted that they could be removed if that information (count by parameter) was not required.


      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
      "Science is about questioning the status quo. Questioning authority".
      The "good enough" maybe good enough for the now, and perfection maybe unobtainable, but that should not preclude us from striving for perfection, when time, circumstance or desire allow.
        Thanks for these sharp comments! I am glad to see perl gurus on this question.

        I will try to implement this structure. Actually I don't need to retain the order between 2.1 and 2.2 for instance. Sorry if it was not clear enough!

        If I use the fetchall_arrayref, maybe it is not worth to change this structure?
        does perl spend a lot of resources to create a strucure?
        Can I come back on this page to add new comments if I need help to implement it or to get some value? There is so much topics posted each day that I assume it is difficult to check if there is a new comment for an old node (except if you look carefully the newest nodes section)

        thanks again.