You said:
I ... am calling the DB about 5 times per run...

Does "one run" mean one command-line execution of the script? How many keys do you generate in one run?

If you are doing lots of runs, part of the slowness will be at the level of the shell, having to build up and tear down a process for each run. Ideally, one run (lasting up to 8 hours or whatever) should minimize this sort of processing overhead.

Apart from that, it's probably more a question of algorithm, and you haven't given us any clues on this. How complicated is the procedure to come up with "normalized keys"? How big does the script really need to be to do this?

And how often do you have to come up with "normalized keys" for a million entries? (Does this need to be done repeatedly? If not, just run it for 8 hours and be done with it -- why worry about timing?)

In reply to Re: Making my Perl program a bit faster by graff
in thread Making my Perl program a bit faster by mrguy123

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.