Sometimes, you can load just one file into memory and then scan the other files one by one and, for each file, line by line, without ever loading the other entire files into memory. And, as a second step, compare the generated files containing the differences between the other files and file 1, which may (or may not) be much smaller than the original files, depending on your data shape.

Another approach (especially if the files are truly huge) is to sort the files according to the comparison key prior to the comparison and then read all of your files line by line in parallel. There is a penalty in sorting the files before the comparison, but it is often worth the cost, because the multifile comparison is then much faster. And, depending on where tour files are coming from, some of them may already be sorted.

Each case is different, so that there is no general strategy blindly applicable to your specific problem, and this is why I can't suggest a solution without knowing in details what you're really comparing and what kind of differences (or common records) you're looking for.

I have multiple fastq files in the following format. I want to print the total count if the second line i.e the sequence matches in all files.

R1.txt

@NS500278:42:HC7M3AFXX:3:21604:26458:18476 2:N:0:AGTGGTCA
AAAAAAAAACAGATATTTGCACTAGGCATTATAAATAACATCAATTAAGTAAAAAAATTA
+
AAAAAEEEEAEEEEEEEEEE/AEEEEEEEEEEEE
     1:R1.txt
[download]

R2.txt

@NS500278:42:HC7M3AFXX:3:21604:26458:18476 2:N:0:AGTGGTCA
AAAAAAAAACAGATATTTGCACTAGGCATTATAAATAACATCAATTAAGTAAAAAAATTA
+
AAAAAEEEEAEEEEEEEEEE
     1:R2.txt
[download]

The output I want is: 

output

@NS500278:42:HC7M3AFXX:3:21604:26458:18476 2:N:0:AGTGGTCA
AAAAAAAAACAGATATTTGCACTAGGCATTATAAATAACATCAATTAAGTAAAAAAATTA
+
AAAAAEEEEAEEEEEEEEEE/AEEEEEEEEEEEE
     1:R1.txt         1:R2.txt      count:2
[download]

My code is:

#!/usr/bin/env perl
use strict;
use warnings;
no warnings qw( numeric );
my %seen;

$/ = "";
while (<>) {
    chomp;
    my ($key, $value) = split ('\t', $_);

    my @lines = split /\n/, $key;
    my $key1 = $lines[1];

    $seen{$key1} //= [ $key ];
    push (@{$seen{$key1}}, $value);
}

foreach my $key1 ( sort keys %seen ) {
my $tot = 0;
my $file_count = @ARGV;
for my $val ( @{$seen{$key1}} ) {
        $tot += ( split /:/, $val )[0];
    }   
    
if ( @{ $seen{$key1} } >= $file_count) {


        print join( "\t", @{$seen{$key1}});
        print "\tcount:". $tot."\n\n";
    }
}
[download]

This is working well with some files but when I compare more files it hangs. I think it is because of memory issue. Without using any modules I want to modify this script so that it can work with any number of files.

Show your existing code.

Also, consider tools like diff, diff3, TortoiseMerge.

Alexander