#!/usr/bin/perl

# https://perlmonks.org/?node_id=1222680

use strict;
use warnings;

while( <DATA> )
  {
  my ($cluster, $member) = split;
  print $cluster eq $member ? "\n" x ($. > 1) : ', ', $member;
  }
print "\n";

__DATA__
Osat_a    Osat_a # just one cluster member
Atha_b    Atha_b # >1 cluster member, this & next line = 2 members
Atha_b    Mtru_c
Fves_d    Fves_d # this & next 2 lines = 3 cluster members
Fves_d    Osat_e
Fves_d    Atha_f
Atha_g    Atha_g # just 1 cluster member
Osat_h    Osat_h
Osat_h    Atha_i
Mtru_j    Mtru_j # just 1 cluster member
[download]

Thank you, tybalt89. Your code worked in terms of generating the expected output.

Could you please explain in detail the following 2 lines from your code? Thank you!

 
my ($cluster, $member) = split;
print $cluster eq $member ? "\n" x ($. > 1) : ', ', $member;
[download]

Let me presume to answer for tybalt89.

my ($cluster, $member) = split;

This depends on default behavior of split, and is equivalent to
    my ($cluster, $member) = split ' ', $_;
The  ' ' split pattern is a special case explained in split docs.

print $cluster eq $member ? "\n" x ($. > 1) : ', ', $member;

This is a bit more tricksy. From the inside out:

• ($. > 1)  $. is input line counter (update: see perlvar).  ($. > 1) evaluates to either '' (empty string) or 1 and will be 1 for every input line after the first.
• "\n" x ($. > 1) Repeats a newline zero times for the first line of input (empty string silently promoted to 0 in this special case), once for every subsequent input line.
• $cluster eq $member ? Newline_or_Nada : ', ' Ternary expression. If  $cluster eq $member true, output newline for every input line after the first (see previous item); if false, output  ', ' string.
• print Ternary_Expression, $member; print result of ternary expresssion (see previous item), then  $member string.
And that's all there is to it (I think).

Update: Minor wording changes.

---

Give a man a fish:  <%-{-{-{-<

In your example input, all the clusters occur contiguously, i.e., all Osat_a members (just the one), then all the Atha_b members, all Fves_d members, etc. Is this the case in your real data, or might you have data like, e.g.,

Osat_a    Osat_a # just one cluster member
Atha_b    Atha_b # >1 cluster member, this & next line = 2 members
Fves_d    Fves_d # this & next 2 lines = 3 cluster members
Osat_h    Osat_h
Atha_b    Mtru_c 
Fves_d    Osat_e
Atha_g    Atha_g # just 1 cluster member
Fves_d    Atha_f
Osat_h    Atha_i
...       ...
[download]

If the former case (all cluster members contiguous) is true, processing of very large files is easy: just buffer all cluster members until you detect the transition from one cluster member to another, then write out all buffered cluster members. This could scale to millions of cluster members.

In the latter case, something like LanX's suggestion seems the way to go.

The input is ordered contiguously. You are correct in your observation

tybalt89's solution processes an input file line-by-line and so has the advantage that it will scale to an input file of any size (well, as long as your HD will hold both the input and output files :).

It seems to me to have the disadvantage of... terseness, shall we say? Let me offer an alternative that is line-by-line and that also:

• Makes a gesture in the direction of input validation. (I strongly believe that time spent on data validation is well spent.)
• Uses easily adapted regexes to validate input data.
• Makes a gesture toward ignoring input that is not of interest.
• Is modular and therefore highly adaptable.
• Incorporates a testing framework for development. (This could be further elaborated by moving the code into its own .pm module and writing a .t file for testing.)
• While being considerably more verbose, is, I would argue, much more maintainable.

c:\@Work\Perl\monks\onlyIDleft>perl process_cluster_info_3.pl
ok 1 - test output
1..1
ok 2 - no warnings
1..2
[download]

Now processing the input, you could use a sliding window to process big chunks (like 100MB), but I don't think it'll make a big difference from reading line by line with readline (since main limitation is hard-disk speed here and Perl and OS are already reading in big chunks behind the scene).

But I would certainly group the write operations, like processing n = 1 million lines before writing out. Collect the entries in a hash of arrays push @{$hash{$cluster}}, $entry and append them to the temporary cluster files ( open has an append mode with '>>' ). Then empty the hash to avoid memory problems and process the next n lines.

NB: In case the entries have to be unique within a cluster (you haven't been precise about that) you'd need a hash of hashes and a more complicated approach.

HTH!

*) I'm not sure about the most efficient way OS-wise to merge large files, but google or the monastery should know. I'm critical about this obsession of you bio-guys of creating huge files. I'd rather have data separated into several smaller files and zipped them together.