Fact 6 : Due to the comparison being done manually by humans, and human making mistakes, we are getting false posetives AND negatives that is significantly impacting our timelines.

Obviously the question is, what should 'ALG' and 'DS' be?

The scenario I have to address :

Where people continue to inspect the diff visually - in this, the performance of the exsiting script is dismal - most of the processing seems to be in sorting the array of lines in lexicographic order (reading/fetching array element : Tie::File::FETCH, Tie::File::Cache::lookup and putting it in it's correct place so that it's sorted : Tie::File::Cache::insert, Tie::File::Heap::insert)

use strict;
use warnings;

use Tie::File;

use Data::Dumper;

use Digest::MD5 qw(md5_hex);

# open an existing file in read-only mode
use Fcntl 'O_RDONLY';

die "Usage: $0 <unsorted input filename> <sorted output filename>" if 
+($#ARGV < 1);

our $recordsWrittenCount = 0;
our $fieldsSorted = 0;

our @array;

tie @array, 'Tie::File', $ARGV[0], memory => 50_000_000, mode => O_RDO
+NLY or die "Cannot open $ARGV[0]: $!";

open(OUTFILE, ">" .  $ARGV[1]) or die "Cannot open $ARGV[1]: $!";

our @tempRecordStorage = ();

our $dx = 0;

# Now read in the EL6 file

our $numberOfLines = @array; # accessing @array in a loop might be exp
+ensive as it is tied?? 

for($dx = 0; $dx < $numberOfLines; ++$dx)
{
    if($array[$dx] eq 'RECORD')
    {
        ++$recordsWrittenCount;
        
        my $endOfRecord = $dx;

        until($array[++$endOfRecord] eq '.')
        {
            push @tempRecordStorage, $array[$endOfRecord];
            ++$fieldsSorted;
        }
        
        print OUTFILE "RECORD\n";

        local $, = "\n";
        print OUTFILE sort @tempRecordStorage;
        @tempRecordStorage = ();

        print OUTFILE "\n.\n"; # PERL does not postfix trailing separa
+tor after the last array element, so we need to do this ourselves)
        
        $dx = $endOfRecord;        
    }
}

close(OUTFILE);

# Display results to user

print "\n[*] Done: " . $fieldsSorted . " fields sorted from " . $recor
+dsWrittenCount . " records written.\n";
[download]

So I thought about it and I believe, some sort if a trie, maybe suffix trie/PATRICIA trie, so that on insertion itself the fields in each record are sorted. Hence, I would not have to sort the final array all in one go and the cost would be amortized (a speculation on my part)

Another issue arises in that case - Tie::File uses array to abstract lines in a file - reading lines into a tree and then serializing them back into an array would require additional memory AND processing/

Question is - would that cost more than the current cost of sorting the tied array?

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.