in reply to Finding multiword units in a corpus

G'day veg_running,

Welcome to the Monastery.

[Aside: Between first reading your post, and subsequently replying, I see you've changed the original. Putting data within <code> tags is good; however, you should indicate the update when doing so after posting. "How do I change/delete my post?" has more about that.]

I used the same tagset as you:

$ cat pm_11148202.tagset.txt udebe <ZUL-SIL-0016-n> ulimi <ZUL-SIL-0017-n> izinyo <ZUL-SIL-0018-n> izinyo lomhlathi <ZUL-SIL-0019-n> ingemuva lomqala <ZUL-SIL-0024-n> umphimbo <ZUL-SIL-0025-n>

You've shown some sample output — this is good; however, you've not shown the source from which that output is derived — this is less good. Also, I see no correlation between the "taglist" tags and the "output" tags. I made up my own sample input data:

$ cat pm_11148202.txt Lokho udebe kukwenze isilomo. Ukuzihlola izinyo kungahlenga izinyo lomhlathi yakho. Amakhala agxiza amafinyila. Ulimi amafutha ulimi wonke ULIMI amabheringi. Sebenzisa amafutha ulimi. Zama ukugwema ukudla okuncinca udebe.

I then ran this code:

#!/usr/bin/env perl use 5.016; use warnings; use autodie; my $corpusname = 'pm_11148202'; my %words2ids; { open my $fh, '<', "$corpusname.tagset.txt"; while (<$fh>) { chomp; my ($text, $token) = split /\t/; $words2ids{fc $text} = $token; } } my $alt = join '|', sort { length($b) <=> length($a) } map fc, keys %words2ids; my $re = qr{(?i:($alt))}; my %found; { open my $in_fh, '<', "$corpusname.txt"; open my $out_fh, '>', "$corpusname.possible-annotation.txt"; while (<$in_fh>) { s/$re/++$found{fc $1}, "$1 $words2ids{fc $1}"/eg; print $out_fh $_; } } delete @words2ids{keys %found}; { open my $fh, '>', "$corpusname.tags-not-found.txt"; for (sort keys %words2ids) { say $fh "$_\t$words2ids{$_}"; } }

This produces

$ cat pm_11148202.possible-annotation.txt Lokho udebe <ZUL-SIL-0016-n> kukwenze isilomo. Ukuzihlola izinyo <ZUL-SIL-0018-n> kungahlenga izinyo lomhlathi <ZUL-S +IL-0019-n> yakho. Amakhala agxiza amafinyila. Ulimi <ZUL-SIL-0017-n> amafutha ulimi <ZUL-SIL-0017-n> wonke ULIMI <ZU +L-SIL-0017-n> amabheringi. Sebenzisa amafutha ulimi <ZUL-SIL-0017-n>. Zama ukugwema ukudla okuncinca udebe <ZUL-SIL-0016-n>.

and

$ cat pm_11148202.tags-not-found.txt ingemuva lomqala <ZUL-SIL-0024-n> umphimbo <ZUL-SIL-0025-n>

Notes:

— Ken

Replies are listed 'Best First'.
Re^2: Finding multiword units in a corpus
by veg_running (Initiate) on Nov 17, 2022 at 12:07 UTC

    Hi Ken, thank you very much for the suggested improvements. I am going to have a look at all of them.

    I since realised I have another problem to solve though: My code only printed the last tag found for a token and not all of them, so if the same token were to appear more than once with a different ID for each occurence in the tagset, I would only get the last one in my output and not a list of all possible IDs assigned to that token.

    Any suggestions on how to handle this would be most appreciated.

      Instead of %words2ids having key-value pairs like

      word => 'id'

      they could perhaps be more like

      word => [qw{id1 id2}]

      I'm not really across the specifics of what you want. You should supply a sample tagset file, a sample input file, and the expected output using those two files.

      My best guess would be changing these lines:

      ... $words2ids{fc $text} = $token; ... s/$re/++$found{fc $1}, "$1 $words2ids{fc $1}"/eg; ...

      to

      ... push @{$words2ids{fc $text}}, $token; ... s/$re/++$found{fc $1}, "$1 @{$words2ids{fc $1}}"/eg; ...

      You should try this for yourself. If you run into difficulties, put together an SSCCE and post it: along with the two sample files and the expected output, this will give us a much better chance of quickly resolving whatever problems you're encountering.

      — Ken

      TIMTOWTDI

      #!/usr/bin/perl use strict; # https://perlmonks.org/?node_id=11148202 use warnings; use List::AllUtils qw( rev_nsort_by ); my $corpusfile = '/tmp/d.11148202.corpus'; # FIXME filename my $wordfile = '/tmp/d.11148202.words'; # FIXME filename my %words2ids; { local @ARGV = $wordfile; while( <> ) { my ($key, $value) = split /[\t\n]/; $words2ids{lc $key} .= " $value"; } } my $pat = do { local $" = '|'; qr/(@{[ map quotemeta, rev_nsort_by { length } keys %words2ids ]})/i}; my %found; { local @ARGV = $corpusfile; print s/\b$pat\K/ $found{lc $1}++; $words2ids{lc $1} /ger while <>; } delete @words2ids{ keys %found }; # not found local $, = "\n"; print '',"---------------- Not Found:", sort(keys %words2ids), '';

      Outputs:

      Lokho udebe <ZUL-SIL-0016-n> kukwenze isilomo. Ukuzihlola izinyo <ZUL-SIL-0018-n> <ZUL-SIL-0018-n-other> kungahlenga +izinyo lomhlathi <ZUL-SIL-0019-n> yakho. Amakhala agxiza amafinyila. Ulimi <ZUL-SIL-0017-n> amafutha ulimi <ZUL-SIL-0017-n> wonke ULIMI <ZU +L-SIL-0017-n> amabheringi. Sebenzisa amafutha ulimi <ZUL-SIL-0017-n>. Zama ukugwema ukudla okuncinca udebe <ZUL-SIL-0016-n>. ---------------- Not Found: ingemuva lomqala umphimbo