The algorithm above requires one to reread the 2Gig files N+1 times. N is the number of records in the file! This is going to take a very long time.

If your files are two gigs you would be better off using a single read of your data. During the first read you would populate a hash. The key of the hash would be something like "gene id|number". The values would be an array that stored all of the sequences associated with that number.

I think you will also find printing out the header and genes much, much easier if you store the data in a hash. If you are concerned about the size of the hash (and at 2Gigs you should be), you might want to use a hash tied to a random access file - search CPAN for Tie::Hash for alternatives.

I also notice that your records are divided by > You can greatly simplify the parsing process by setting the record separator variable $/ (see perlvar for details). If you set $/='>', then you can read in an entire record at once. You won't need to figure out whether the current line is the header or the list of sequences.

The following code sample illustrates one way you might use $/ and a hash: 

#use local so that the setting doesn't interfere with
#other places you might want to read in data.
local $/='>';

#load sequences
my %hSequences;
while (my $line = <DATA>) {
  chomp $line;
  next if $line eq '';

  #extract data from record
  #  -- s at end of regex needed so that . matches new lines
  my ($subs, $gid, $sSequences)
    = $line
      =~ /^(\d+)_\d+\s+geneid(\d+)\s+\d+\s+\d+\slen=\d+\s+(.*)$/s;

  #populate hash
  my $sKey="$subs $gid";
  my $aSequences = $hSequences{$sKey};
  $hSequences{$sKey} = $aSequences = []
    unless defined($aSequences);
  push @$aSequences, split(/\s+/, $sSequences);
}

#print results
while (my ($k, $v) = each(%hSequences)) {
  $k =~ s/ / gid=/;
  print ">$k\n" . join("\n", sort @$v) . "\n";
}

[download]

Best, beth

In reply to Re: array of arrays - printing data by ELISHEVA
in thread array of arrays - printing data by sugar

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.