Once all this is established, i start itterating the input file (which can be millions and millions of lines), and sending the filenames over the ssh connection to this remote script, then it waits for a response. when it gets a response, it sends the next line

You are peforming a huge number of roundtrips of small packets over the network between collaborating processes and your processes over the network have to wait for each other's input. This is notoriously slow (just try it over a wide area network to get a feel for the impact of the roundtrips in general). Even while the network itself is fast the fact that your processes have to wait for roundtrip results will slow down the entire operation.

Therefore your algorithm may benefit greatly from transfer of larger chunks in one shot over the network. In particular I would suggest to copy the searchlist to the nodes (http/ftp-like protocol, Perl module LWP), then execute a fully local search (stdin from the local search list file, stdout to a local result file on the same node) and only afterwards transfer the result file back in one shot to the central system for report.

In reply to Re: Searching a distributed filesystem by varian
in thread Searching a distributed filesystem by LostShootingStar

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.