If/when you get to a point of having many millions of keys in your database, the start-up time to load all of them into memory for your "fast check", and (eventually) the memory consumption for the hash itself, could put you beyond a point of diminishing returns.

Depending on how big the DB table gets, how many look-ups you actually do in one run of your script, and what else is going on besides look-ups, you might cross a threshold where the script runs faster if you just do queries to check for key values, rather than loading all keys into a hash for that purpose.

BTW, if the reason for checking the existence of a key is to decide whether you should do an insert vs. an update (or insert vs. nothing), you might want to check out the "INSERT ... ON DUPLICATE KEY UPDATE ..." syntax in mysql.

In any case, if speed is really an issue, you'll want to have a benchmark for testing the alternatives. Use Benchmark if you like, or just have two versions of a job that will do a fair test for both approaches. You'll want it to be able to compare the timing now, and also make an equivalent comparison at any time in the future, to see if table size affects one approach differently from the other.

In reply to Re: efficiency of exists() by graff
in thread efficiency of exists() by MrAtheist

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.