Compare two files with Japanese and English data

solukas has asked for the wisdom of the Perl Monks concerning the following question:

Dear all,

I am a newbie of Perl. I am asked to compare two files which contain some Japanese and English informaion. I use a slow approach, read two files and put them in two arrays, then use while loop to check whether the File_B's TERMS are WITHIN the lines of File_A.

File_A:

This is a boy.

ハンカ

She is a girl.

I am a man.

This apple is big.

That orange is small.

File_B:

is a

ハンカ

small

Output:

I am a man.

This apple is big.

The program I wrote works fine. However, the files I am processing are very huge. The running time is so slow if I use looping like this. My friend told me that I can put the files into hash and it will run far much quicker. I can see the point of using it but the files I have are random and also the terms in File_B can be in any part of File_A. Some previous posts suggest that I can use split the lines of File_A, put the line in hash and compare it with File_B (put File_B in the other hash as well). However, I do not know how to do it besides using ~// to check whether the line got the term.

Thanks very much for your help indeed!

Kind regards,

Luke

open(A_FILE, "<", "FILE_A.txt");

my(@a_lines) = <A_FILE>; # read file into list

close(A_FILE);

open(B_FILE, "<", "FILE_B.txt");

my(@b_lines) = <B_FILE>; # read file into list

my($b_lines);

close(B_FILE);

open(my $out, ">",  "Useful.txt") or die "Can't open Useful.txt: $!";

$number = @b_lines;

foreach $a_line (@a_lines) # loop thru list

{

    $found = 0;

    my $sentence = $a_line;

    $i = 0;

    chomp($sentence);

    while (($i <= $number-1) and ($found == 0)){

    chomp($b_lines[$i]);

    if ($sentence =~ /$b_lines[$i]/){

        $found = 1;

    }

    $i++;

    }

    if ($found == 1) {

        print $out $sentence."\n";

    }

}
[download]

Comment on Compare two files with Japanese and English data Download Code

Replies are listed 'Best First'.
Re: Compare two files with Japanese and English data by GrandFather (Saint) on Apr 15, 2011 at 23:24 UTC
Use a constructed regular expression to perform the matching on the keys: use strict; use warnings; use encoding 'UTF8'; my $aData = <<ADATA; This is a boy. ハンカ She is a girl. I am a man. This apple is big. That orange is small. ADATA my $bData = <<BDATA; is a ハンカ small BDATA open my $inB, "<:utf8", \$bData; my (@keys) = <$inB>; close ($inB); chomp @keys; my $match = join '\|', sort {length $b <=> length $a} @keys; $match = qr{($match)}; open my $inA, "<:utf8", \$aData; while (defined (my $line = <$inA>)) { print "Found '$1' in line $.:\n $line" if $line =~ $match; } [download] Prints: `Found 'is a' in line 1: This is a boy. Found 'ハンカ' in line 2: ハンカ Found 'is a' in line 3: She is a girl. Found 'is a' in line 5: This apple is big. Found 'small' in line 6: That orange is small.` [download] Note: I don't seem to be able to paste ハンカ into the code blocks. Replace the entities with the actual string for correct behaviour. True laziness is hard work	[reply] [d/l] [select]

Replies are listed 'Best First'.

Re: Compare two files with Japanese and English data
by GrandFather (Saint) on Apr 15, 2011 at 23:24 UTC

Use a constructed regular expression to perform the matching on the keys:

use strict;
use warnings;
use encoding 'UTF8';

my $aData = <<ADATA;
This is a boy.
&#12495;&#12531;&#12459;
She is a girl.
I am a man.
This apple is big.
That orange is small.
ADATA

my $bData = <<BDATA;
is a
&#12495;&#12531;&#12459;
small
BDATA

open my $inB, "<:utf8", \$bData;
my (@keys) = <$inB>;
close ($inB);
chomp @keys;

my $match = join '|', sort {length $b <=> length $a} @keys;
$match = qr{($match)};

open my $inA, "<:utf8", \$aData;
while (defined (my $line = <$inA>)) {
    print "Found '$1' in line $.:\n   $line" if $line =~ $match;
}
[download]

Prints:

Found 'is a' in line 1:
   This is a boy.
Found '&#12495;&#12531;&#12459;' in line 2:
   &#12495;&#12531;&#12459;
Found 'is a' in line 3:
   She is a girl.
Found 'is a' in line 5:
   This apple is big.
Found 'small' in line 6:
   That orange is small.
[download]

Note: I don't seem to be able to paste ハンカ into the code blocks. Replace the entities with the actual string for correct behaviour.

True laziness is hard work

[reply]
[d/l]
[select]