I agree that the data should go to the node's scratch, the processing happens, and results are uploaded to NFS again. Never mind NFS, RAID5 is also not helping writes. However, sometimes files become so ridiculously large that the local scratch does not suffice, and I am forced to work off NFS - though I still try to put the products on scratch, and upload them to NFS afterwards.

I did write a simple FullyBuffered module basically doing sysopen, sysread into a large buffer, maintaining a cursor (to avoid unecessary string copies), etc. I was timing this against the original script using Perl's IO, and it performed, to my surprise, a little worse. The Perl IO version takes about 3 minutes for 2^20 lines, my fully buffered approached 4 minutes for 2^20 lines (dang, I tossed its log file). Of course, I do suspect that I am doing something stupidly inefficient.

I thank you very much for showing me the proper utf8 conversions. Should I continue with my module approach, it will come in handy.

In reply to Re^2: 4k read buffer is too small by voeckler
in thread 4k read buffer is too small by voeckler

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.