Hi I written a perl to post process 2 files data and then put it as output file. However, each of the file containt over 8millions of line. My script seem effective at all as after several hours, it only reach thousand line. I am here seeking help that anyone could fix my script to a better ways.

The logic basically just take file 1 first 2 column data match with file 2. If it is match, then insert all the 3 colume from file 1 to the file 2. Also, the nature of file 1 and file 2 already sorted correctly (that meant line 1 in file1 is equal to line1 in file2. it no need to perform search actually)

 
File 1 example:- 

WL,BL,Die1
WL0,BL0,1708
WL0,BL1,1708
WL0,BL2,1708
WL0,BL3,1931
WL0,BL4,1931

File 2 example:- 

WL,BL,Die 2
WL0,BL0,1708
WL0,BL1,1931
WL0,BL2,1708
WL0,BL3,1931
WL0,BL3,1708

Output after script:- 

WL,BL,Die1, Die2
WL0,BL0,1708,1708
WL0,BL1,1708,1931
WL0,BL2,1708,1708
WL0,BL3,1931,1931
WL0,BL4,1931,1708

My script:- 

#!/usr/bin/perl 

#Copy Die2 as Output file
use File::Copy; 
copy ("Die2_10k.txt","CombineDie1Die2.txt") or die "copy failed: $!"; 

#Open Die1 Input file 
open (Label, "Die1_10k.txt") or die "can't open Die1: $!"; 

#Search and replace using 1 liner command 
while (<Label>) { 
    $replace = $_; 
    chomp ($replace); 
    @temp = split (/,/, $replace); 
    $search = $temp[0].",".$temp[1];  
    
    $command_line = "perl -pi\.bak -e s\/".$search."(?=,)\/".$replace.
+"\/g\; CombineDie1Die2.txt"; 
    
    system ($command_line); 
}
[download]

It will be greatful if someone could help to provide a effective way of process the 2 8millions of file with some script for the above purpose. Thanks you.

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.