Hi!

Excellent analysis, thanks!

Well I will try out all your suggestions and complete code later today, but one MAJOR change I did was to just change the output file from using Tie::File to normal file IO (using open)

.. and the performance got multiplied by approx 7 - 10 times.

This of course becomes apparent once you look at the profiler output I pasted above.

I ran the profiler on a file having some 14k "records" (839754 lines) and the tmon results: 

Total Elapsed Time = 456.5656 Seconds
  User+System Time = 206.2156 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c  Name
 20.2   41.73 41.732 167950   0.0000 0.0000  Tie::File::_read_record
 14.0   28.96 154.93 157920   0.0000 0.0001  Tie::File::_fetch
 12.4   25.71 180.65 157920   0.0000 0.0001  Tie::File::FETCH
 10.2   21.16 35.610 157920   0.0000 0.0000  Tie::File::Cache::lookup
 7.16   14.77 53.769 839753   0.0000 0.0001  Tie::File::Cache::insert
 5.64   11.62 12.893 839755   0.0000 0.0000  Tie::File::_seek
 5.60   11.54 38.998 839753   0.0000 0.0000  Tie::File::Heap::insert
 5.33   10.99 10.991 839753   0.0000 0.0000  Tie::File::Cache::_heap_m
+ove
 5.13   10.57 23.874 839753   0.0000 0.0000  Tie::File::Heap::_insert_
+new
 3.30   6.796  9.688 739447   0.0000 0.0000  Tie::File::Heap::promote
 3.25   6.701 24.732      1   6.7008 24.731  Tie::File::_fill_offsets
 3.14   6.474  6.474 157920   0.0000 0.0000  Tie::File::Heap::_nseq
 2.31   4.756 14.444 739447   0.0000 0.0000  Tie::File::Heap::lookup
 1.12   2.311  2.311 839753   0.0000 0.0000  Tie::File::Heap::_nelts_i
+nc
 0.62   1.271  1.271 839760   0.0000 0.0000  Fcntl::__ANON__

[download]

In reply to Re^2: Improving dismal performance - Part 1 by PoorLuzer
in thread Improving dismal performance - Part 1 by PoorLuzer

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.