TIMTOWTDI 

#!/usr/bin/perl

use strict; # https://perlmonks.org/?node_id=11148202
use warnings;
use List::AllUtils qw( rev_nsort_by );

my $corpusfile = '/tmp/d.11148202.corpus'; # FIXME filename
my $wordfile = '/tmp/d.11148202.words'; # FIXME filename

my %words2ids;
{
local @ARGV = $wordfile;
while( <> )
  {
  my ($key, $value) = split /[\t\n]/;
  $words2ids{lc $key} .= " $value";
  }
}
my $pat = do { local $" = '|'; qr/(@{[
  map quotemeta, rev_nsort_by { length } keys %words2ids ]})/i};
my %found;
{
local @ARGV = $corpusfile;
print s/\b$pat\K/ $found{lc $1}++; $words2ids{lc $1} /ger while <>;
}

delete @words2ids{ keys %found }; # not found
local $, = "\n";
print '',"---------------- Not Found:", sort(keys %words2ids), '';

[download]

Outputs: 

Lokho udebe <ZUL-SIL-0016-n> kukwenze isilomo.
Ukuzihlola izinyo <ZUL-SIL-0018-n> <ZUL-SIL-0018-n-other> kungahlenga 
+izinyo lomhlathi <ZUL-SIL-0019-n> yakho.
Amakhala agxiza amafinyila.
Ulimi <ZUL-SIL-0017-n> amafutha ulimi <ZUL-SIL-0017-n> wonke ULIMI <ZU
+L-SIL-0017-n> amabheringi.
Sebenzisa amafutha ulimi <ZUL-SIL-0017-n>.
Zama ukugwema ukudla okuncinca udebe <ZUL-SIL-0016-n>.

---------------- Not Found:
ingemuva lomqala
umphimbo

[download]

In reply to Re^3: Finding multiword units in a corpus by tybalt89
in thread Finding multiword units in a corpus by veg_running

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.