Re^2: Finding multiword units in a corpus

Hi Ken, thank you very much for the suggested improvements. I am going to have a look at all of them.

I since realised I have another problem to solve though: My code only printed the last tag found for a token and not all of them, so if the same token were to appear more than once with a different ID for each occurence in the tagset, I would only get the last one in my output and not a list of all possible IDs assigned to that token.

Any suggestions on how to handle this would be most appreciated.

Comment on Re^2: Finding multiword units in a corpus

Replies are listed 'Best First'.
Re^3: Finding multiword units in a corpus by kcott (Archbishop) on Nov 17, 2022 at 16:19 UTC
Instead of `%words2ids` having key-value pairs like `word => 'id'` [download] they could perhaps be more like `word => [qw{id1 id2}]` [download] I'm not really across the specifics of what you want. You should supply a sample tagset file, a sample input file, and the expected output using those two files. My best guess would be changing these lines: `... $words2ids{fc $text} = $token; ... s/$re/++$found{fc $1}, "$1 $words2ids{fc $1}"/eg; ...` [download] to `... push @{$words2ids{fc $text}}, $token; ... s/$re/++$found{fc $1}, "$1 @{$words2ids{fc $1}}"/eg; ...` [download] You should try this for yourself. If you run into difficulties, put together an SSCCE and post it: along with the two sample files and the expected output, this will give us a much better chance of quickly resolving whatever problems you're encountering. — Ken	[reply] [d/l] [select]
Re^3: Finding multiword units in a corpus by tybalt89 (Monsignor) on Nov 18, 2022 at 16:55 UTC
TIMTOWTDI #!/usr/bin/perl use strict; # https://perlmonks.org/?node_id=11148202 use warnings; use List::AllUtils qw( rev_nsort_by ); my $corpusfile = '/tmp/d.11148202.corpus'; # FIXME filename my $wordfile = '/tmp/d.11148202.words'; # FIXME filename my %words2ids; { local @ARGV = $wordfile; while( <> ) { my ($key, $value) = split /[\t\n]/; $words2ids{lc $key} .= " $value"; } } my $pat = do { local $" = '\|'; qr/(@{[ map quotemeta, rev_nsort_by { length } keys %words2ids ]})/i}; my %found; { local @ARGV = $corpusfile; print s/\b$pat\K/ $found{lc $1}++; $words2ids{lc $1} /ger while <>; } delete @words2ids{ keys %found }; # not found local $, = "\n"; print '',"---------------- Not Found:", sort(keys %words2ids), ''; [download] Outputs: `Lokho udebe <ZUL-SIL-0016-n> kukwenze isilomo. Ukuzihlola izinyo <ZUL-SIL-0018-n> <ZUL-SIL-0018-n-other> kungahlenga +izinyo lomhlathi <ZUL-SIL-0019-n> yakho. Amakhala agxiza amafinyila. Ulimi <ZUL-SIL-0017-n> amafutha ulimi <ZUL-SIL-0017-n> wonke ULIMI <ZU +L-SIL-0017-n> amabheringi. Sebenzisa amafutha ulimi <ZUL-SIL-0017-n>. Zama ukugwema ukudla okuncinca udebe <ZUL-SIL-0016-n>. ---------------- Not Found: ingemuva lomqala umphimbo` [download]	[reply] [d/l] [select]