You could save some memory consumption (and perhaps reduce slowdown due to memory paging) if you opened the files one by one. By opening all the files at once, you are also opening 200 buffers for each of the files all at once.

  1. Create an array of arrays.
  2. Read the first file one line at a time, and place all of column 2's values in $aa[0];
  3. Read the second file one line at a time, and place all of column 2's values in $aa[1];
  4. And so on...
  5. Loop through array and print to file

With files that large, you might also want to consider tying the arrays holding the columns to random access files. I once had a ten-fold speed up (in a C++ program) just by using temp files instead of RAM to store data while I was processing. See Tie::File

If your columns are fixed width, you might be able to to avoid two cycles, one to read the array and one to print the array, by keeping a variable that stores the current line length. Then instead of an array of arrays, you could just:

  1. Read the first file
    1. Write the first file by placing each col 2 value on a separate line.
    2. Increase the line length by one column width
  2. Read the next file
    1. Seek to the end of the line, insert the column
    2. Seek to end of the column you just wrote
    3. Seek to end of next line, insert the column
    4. After all lines have been read, increase line length by one column width
  3. Repeat until all files are processed

Best, beth

In reply to Re: Merging Many Files Side by Side by ELISHEVA
in thread Merging Many Files Side by Side by sesemin

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.