in reply to Re^3: Ordering Template cards
in thread Ordering Template cards

I always use refs as well. My first question at an API is if it can provide me with refs. Provided that the returned data will be processed locally and immediately

Replies are listed 'Best First'.
Re^5: Ordering Template cards
by Marshall (Canon) on Jan 28, 2021 at 06:30 UTC
    I could be wrong about this, but in my testing, asking for an array_ref like above, causes the entire result set to be copied into Perl memory. If you undefine the array_ref or assign it another value, Perl can reuse that memory. I haven't benchmarked this, but I'm sure that other Monks have: I suspect that getting an array_ref is faster than going row by row with the exception of an enormous result set.
[reply]

      Marshall, that's a good point regarding db-operations. You are right that there is memory cost for the performance boost of using refs with this specific use-case: db-IO. I was talking more generally so probably I gave the wrong impression for this specific case. Anyway refreshing on the manual is good.

      According to fetchall_arrayref of DBI a tradeoff would be to fetchall_arrayref() in batches but still use array-refs, as

      That might be the fastest way to fetch and process lots of rows using the DBI, but it depends on the relative cost of method calls vs memory allocation.
      my $rows = []; # cache for batches of rows
while( my $row = ( shift(@$rows) || # get row from cache, or reload ca
+che:
                   shift(@{$rows=$sth->fetchall_arrayref(undef,10_000)
+||[]}) )
) {
  ...
}

      [download]

      bw, bliako

[reply]
[d/l]
[select]
        bliako, Most excellent reference!

        I am an "old school hardware guy", dating back to the dark ages where 64K bytes of iron core memory was a lot! My current machine is 64 bits with 16 GB of semiconductor memory. Memory usage is something to be aware of, but I don't worry about it often, but "seldom does not mean never".

        I suppose that the unwary could get into a "memory thrashing" situation with a huge allocation of virtual memory. With my experience so far, if the result set is so humongous that it won't fit into physical memory, then there may be a problem with the SQL query itself! That is something to be considered also!

[reply]

In Section Seekers of Perl Wisdom