I have done this type of thing a lot on Solaris boxes, I need to know a few more bits of data from you to give you a fast working solution:

1, What type of sort are you using? Are you sorking on multiple "keys"? If so are all the sorts the same direction (all accending or mix and match). Give me an example from your posted dataset.

2, I need to know what your system usage looks like can you dump the second display of mpstat 1 2 while your perl version is running.

Let me know and I will be able to help.

Edited, I missed some info on your post, here is what I got just running a test on my devel Sun420r 2 proc 2gb ram box: 24 mil records in 20.25 minutes. Here is how I did it:
/usr/bin/sort -k1 -k2 -T /data2 -o t.sorted t
this says "sort on the first field then if there are duplicates sort on the second, use /data2 as a temp file area and output to t.sorted instead of stdout. You will need 4x <size of inputfile> space free on the temp working dir. Also your sortrate may be better than mine cause I had a lot of duplicate records which forced the secondary key sort to happen way more frequently. If you need to sort on 40000600045 first and then break ties with the first field use -k2 -k1. Let me know if you have questions.

-Waswas

In reply to Re: Fast/Efficient Sort for Large Files by waswas-fng
in thread Fast/Efficient Sort for Large Files by mjmaresca

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.