I would have preferred something that concats the entirety of the genome and then shuffles and fragments as per length distribution of chromosomes in input

Why? Why not consume only as much data as required per current "length" (btw, are they really constant (1e6)?), and then dispose of data no longer needed as you move along? Reversing every second fragment before shuffling is kind of mad science... (well, in positive sense)

Here's a take, that, while "concatenating in entirety", tries hard not to allocate any more memory. And even less if id_lines can be re-built during output, i.e. not stored. Though I didn't profile.

The problem I found interesting for myself to look into was this: if there's huge chunk of bytes to, e.g., shuffle, why split or make copies or construct huge Perl lists etc. I hope code below shuffles "in place". C code is more or less ripped from PDL::API. It's puzzling to me why is it that "wrapping existing data into piddle in-place" exists (for 20+ years?) as somewhat obscure part of documentation and not implemented in pure Perl.

The RNG ('taus') was chosen arbitrarily because of synopsis, there are plenty of others to choose from, more fun than simple "fragment reversing", so have a look :)

use strict;
use warnings;
use PDL;
use Inline with => 'PDL';
use PDL::GSL::RNG;

my $genome = '';
my @id_lines;
my @runs;

########################## 
# Read 
########################## 

while ( <DATA> ) { 
    chomp;
    if ( /^>/ ) {
        push @id_lines, $_
    }
    else {
        push @runs, length;
        $genome .= $_
    }
}

########################## 
# Shuffle
########################## 

my $rng = PDL::GSL::RNG-> new( 'taus' );
$rng-> set_seed( time );

my $start  = 0;
my $stop   = length( $genome ) - 1;
my $window = 3;

while ( $start < $stop ) {
    my $len = $start + $window > $stop
        ? $stop - $start
        : $window;
    my $p = mkpiddle( $genome, $start, $len );
    $rng-> ran_shuffle( $p );
    $start += $window
}

########################## 
# Output
########################## 

$start = 0;
for ( 0 .. $#runs ) {
    print $id_lines[ $_ ], "\n",
          substr( $genome, $start, $runs[ $_ ]), "\n";
    $start += $runs[ $_ ]
}

########################## 
# Guts
########################## 
    
use Inline C => <<'END_OF_C';

static void default_magic( pdl *p, int pa ) { p-> data = 0; }

pdl* mkpiddle( char* data, int ofs, int len ) {
    PDL_Indx dims[] = { len };

    pdl* npdl = PDL-> pdlnew();
    
    PDL-> setdims( npdl, dims, 1 );
    npdl-> datatype = PDL_B;
    npdl-> data     = data + ofs;
    npdl-> state   |= PDL_DONTTOUCHDATA | PDL_ALLOCATED;
    PDL-> add_deletedata_magic( npdl, default_magic, 0 );
    return npdl;
}

END_OF_C

__DATA__
>Chr1
CCCTAAACCCTAAACCCTAAACCCTAAACCTCTGAATCCTTAATCCCTAAATCCCTAAAT
>Chr2
TATGACGTTTAGGGACGATCTTAATGACGTTTAGGGTTTTATCGATCAGCGACGTAGGGA
>Chr3
GTTTAGGGTTTAGGGTTTAGGGTTTAGGGTTTAGGGTTTAGGGTTTAGGGTTTAGGGTTT
>Chr4
AACAAGGTACTCTCATCTCTTTACTGGGTAAATAACATATCAACTTGGACCTCATTCATA
>Chr5
AACATGATTCACACCTTGATGATGTTTTTAGAGAGTTCTCGTGTGAGGCGATTCTTGAGG

[download]

In reply to Re^4: Reduce RAM required by vr
in thread Reduce RAM required by onlyIDleft

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.