Fact 6 : Due to the comparison being done manually by humans, and human making mistakes, we are getting false posetives AND negatives that is significantly impacting our timelines.

Obviously the question is, what should 'ALG' and 'DS' be?

The scenario I have to address :

Where people continue to inspect the diff visually - in this, the performance of the exsiting script is dismal - most of the processing seems to be in sorting the array of lines in lexicographic order (reading/fetching array element : Tie::File::FETCH, Tie::File::Cache::lookup and putting it in it's correct place so that it's sorted : Tie::File::Cache::insert, Tie::File::Heap::insert)

use strict;
use warnings;

use Tie::File;

use Data::Dumper;

use Digest::MD5 qw(md5_hex);

# open an existing file in read-only mode
use Fcntl 'O_RDONLY';

die "Usage: $0 <unsorted input filename> <sorted output filename>" if 
+($#ARGV < 1);

our $recordsWrittenCount = 0;
our $fieldsSorted = 0;

our @array;

tie @array, 'Tie::File', $ARGV[0], memory => 50_000_000, mode => O_RDO
+NLY or die "Cannot open $ARGV[0]: $!";

open(OUTFILE, ">" .  $ARGV[1]) or die "Cannot open $ARGV[1]: $!";

our @tempRecordStorage = ();

our $dx = 0;

# Now read in the EL6 file

our $numberOfLines = @array; # accessing @array in a loop might be exp
+ensive as it is tied?? 

for($dx = 0; $dx < $numberOfLines; ++$dx)
{
    if($array[$dx] eq 'RECORD')
    {
        ++$recordsWrittenCount;
        
        my $endOfRecord = $dx;

        until($array[++$endOfRecord] eq '.')
        {
            push @tempRecordStorage, $array[$endOfRecord];
            ++$fieldsSorted;
        }
        
        print OUTFILE "RECORD\n";

        local $, = "\n";
        print OUTFILE sort @tempRecordStorage;
        @tempRecordStorage = ();

        print OUTFILE "\n.\n"; # PERL does not postfix trailing separa
+tor after the last array element, so we need to do this ourselves)
        
        $dx = $endOfRecord;        
    }
}

close(OUTFILE);

# Display results to user

print "\n[*] Done: " . $fieldsSorted . " fields sorted from " . $recor
+dsWrittenCount . " records written.\n";
[download]

So I thought about it and I believe, some sort if a trie, maybe suffix trie/PATRICIA trie, so that on insertion itself the fields in each record are sorted. Hence, I would not have to sort the final array all in one go and the cost would be amortized (a speculation on my part)

Another issue arises in that case - Tie::File uses array to abstract lines in a file - reading lines into a tree and then serializing them back into an array would require additional memory AND processing/

Question is - would that cost more than the current cost of sorting the tied array?