Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

Re: Private Utilities

by l3v3l (Monk)
on Dec 01, 2005 at 18:05 UTC ( #513370=note: print w/replies, xml ) Need Help??


in reply to Private Utilities

Perfect! I have been looking for just such a solution this morn and appreciate your post - here is one I just put together really quick ... to chunk up a given fasta file (unaligned sequences) in to N files for submission of multiple smaller jobs on a cluster:
#!/usr/bin/perl -w use strict; my $num_ch = $ARGV[0] or die "must pass the # of chunks $!"; open (FH,"<$ARGV[1]") or die "must pass the file name to make chunks +of $!"; my @lines = (); $/ = ">"; while (<FH>){ chomp; next if /^$/; # get rid of blank lines push @lines,">$_"; } close FH; my $num_rec = scalar(@lines); print "number of records : $num_rec\n"; #die "Chunks exceed records!!" unless ($num_rec >= $num_ch); sub write_em { my $output = shift()."_chunk.fa"; my $ar_ref = shift; open (QRTR,">$output") or die "Cannot open $output : $!"; print QRTR @$ar_ref; close (QRTR) or die "cannot close $output : $!"; } my $cnt = 0; my $rng = int($num_rec/$num_ch) + ($num_rec%$num_ch ? 1 : 0); for (1..$num_ch) { write_em($_,[@lines[$cnt..($_ == $num_ch ? $#lines : ($cnt+($r +ng-1)))]]); $cnt += $rng; }
Feel free to comment or point me in the right direction where I have gone astray - this just scratched a specific itch ...

Replies are listed 'Best First'.
Re^2: Private Utilities
by thor (Priest) on Dec 01, 2005 at 20:06 UTC
    From what I understand, these files can be large. I also see that you're essentially reading the whole file into memory. The combination of those two could be bad. Before I propose a better solution, is it okay to put adjacent lines in different files? That is, let's say that I have a file that contains the following:
    AAA BBB CCC DDD EEE FFF
    Is it okay to split into two files like so:
    AAA CCC EEE --- BBB DDD FFF
    If so, I may have a really slick solution floating in my head...

    thor

    The only easy day was yesterday

      no problem!! round robin is fine - here is a way I have done it in a liner without having to read the entire file into mem (just an example) :
      # Choose first N FASTA records perl -ne 'BEGIN {$/=">";$o=0}{chomp;$o<N?(/^$/?next:print">$_"):last;$ +o++}' EXAMPLE.fa
      example usage:
      perl -ne 'BEGIN{$/=">";$o=0}{chomp;$o<22?(/^$/?next:print">$_"):last; +$o++}' b4x_est100.fa # print the first 22 records
        Alright...here's a simple multiplexer:
        use strict; use warnings; use Getopt::Long; my ($number, $format) = (2, "output"); GetOptions( "number=i" => \$number, "format=s" => \$format, ); my @output_file; foreach my $num (1..$number) { my $file = "$format.$num"; open( $output_file[$num-1], ">", $file ) or die "Couldn't open + '$file' for write: $!\n"; } my $file_num = 0; while(<>) { print {$output_file[$file_num]} $_; $file_num += 1; $file_num %= $number; }
        Save it in a file of your choosing, and call it like:
        perl script.pl --format out --number 7 input1 input2 ...

        thor

        The only easy day was yesterday

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://513370]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (5)
As of 2022-05-17 15:15 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Do you prefer to work remotely?



    Results (66 votes). Check out past polls.

    Notices?