in reply to Splitting compound (concatenated) words )

If all the words are correctly spelt and are in your dictionary, then this appears to make a good attempt at many inputs: 

#! perl -slw
use strict;

my @w = do{ local @ARGV = 'words.txt'; <> };
chomp @w;

my $s = 'couldsomeonerecommendaworkingperlmoduletosplitconcatenatedwor
+ds';

my @subset = grep{ $s =~ /$_/ } 'a', 'perl', @w;

my $re1 = join '|', sort{ length( $b ) <=> length( $a ) }@subset;
my $re2 = "($re1)?" x 11;

print for grep defined(), $s =~ /^$re2$/;

__END__
C:\test>junk
could
someone
recommend
a
working
perl
module
to
split
concatenated
words

[download]

But note: I had to add 'a' & 'perl' which don't appear in my dictionary; and I cheated by hardwiring the number of words (11) to look for.

If I change that to a larger number (say 100), then the results are less good:

C:\test>junk
could
someone
recommend
aw
or
king
perl
module
to
split
concatenated
words

[download]

However, if I removed the iffy non-word 'aw' from my rather permissive dictionary, it once again produces the right output, but that just goes to prove how sensitive and dependent the results would be on a good dictionary and correct spelling.

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

The start of some sanity?

Replies are listed 'Best First'.
Re^2: Splitting compound (concatenated) words )
by vit (Friar) on May 15, 2012 at 23:49 UTC
    Thanks, great code. This is really something and I will probably take advantage of this.

    However I would prefer to have some probabilistic approach so that no need in guessing between 11 or 100. There should be some NLP related module, I think.
[reply]
      However I would prefer to have some probabilistic approach so that no need in guessing between 11 or 100.

      The value of 100 is just an arbitrary value that just needs to be large enough to encompass the number of words in the input.

      I'm not sure you'd get much better results with NLP, but its been a long time since I did anything with that and the art moves on. Good luck.

      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.

      The start of some sanity?
[reply]
Re^2: Splitting compound (concatenated) words )
by vit (Friar) on May 16, 2012 at 22:06 UTC
    This is not working for me.
    I do not understand how 
    $s =~ /^$re2$/;

    [download]
    can match from the beginning to the end for 
    my $s = 'couldsomeonerecommendaworkingperlmoduletosplitconcatenatedwor
+ds';

    [download]
    and 
    ## $re2 = (recommend|someone|working|module|words|split|comme|could|..
+.

    [download]
[reply]
[d/l]
[select]
      I do not understand how $s =~ /^$re2$/; can match from the beginning to the end for my $s = 'couldsomeonerecommendaworkingperlmoduletosplitconcatenatedwords'; and ## $re2 = (recommend|someone|working|module|words|split|comme|could|...

      If that's what $re2 looks like, then you are not running the code I posted!

      It should look like:

      (concatenated|concatenate|catenated|recommend|catenate|commend|someone
+|working|catena|module|could|enate|split|words|perl|cate|king|mend|so
+me|word|work|ate|cat|con|daw|end|eon|ere|kin|let|lit|men|mod|one|per|
+rec|som|ted|ten|at|aw|ed|en|er|et|in|it|ki|li|me|mm|mo|na|ne|od|om|on
+|or|os|pe|re|so|to|wo|a)?
... repeat 11 (or more) times.

      [download]

      The way the code:

      #! perl -slw
use strict;

my @w = do{ local @ARGV = 'words.txt'; <> };
chomp @w;

my $s = 'couldsomeonerecommendaworkingperlmoduletosplitconcatenatedwor
+ds';
my @subset = grep{ $s =~ /$_/ } 'a', 'perl', @w;
my $re1 = join '|', sort{ length( $b ) <=> length( $a ) }@subset;
my $re2 = "($re1)?" x 11;

print for grep defined(), $s =~ /^$re2$/;

      [download]

      Works is:

      1. First it loads the dictionary into @w.
      2. Then it filters that array against the input string with grep, to produce @subset

        This is the subset of all words in the dictionary that appear somewhere in the input string.

      3. It then build $re1, which is an alternation of all those words, longest first.

        So $re1 looks like this: longestword|longword|shorter|short

      4. It then builds $re2 which looks like this:
        (longestword|longword|shorter|short)?(longestword|longword|shorter|sho
+rt)?(longestword|longword|shorter|short)?...

        [download]

        The effect is that (provided every word in your input is spelt correctly, and appears in the dictionary), is that it will match each longest word in turn.

        Because the repeated regex is conditional ($re1)?, if the number of repeated elements is longer than the number of words in the string, it will just stop matching when it reaches the end of the string, and the redundant captures will return the null string.

        Hence the grep defined().

      The results will rarely be perfect, but it depending upon the source of your strings, it might form the basis for further, perhaps statistical, analysis.

      I'm curious as hell about the source of the data and the purpose of the exercise?

      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.

      The start of some sanity?
[reply]
[d/l]
[select]
        It actually works. I created a dictionary of 50000+ words sorted by popularity. The problem was that I ran your example but I did not have "concatenated" in there.
        Thanks for your explanation. I will spend some time to understand this part 
        print for grep defined(), $s =~ /^$re2$/;

        [download]
        in details. Looks like genius. In terms of both, Perl flexibility and your implementation. Actually it needs 
        print "$_\n" for grep defined(), $s =~ /^$re2$/;

        [download]
        to print line-wise.

        I'm curious as hell about the source of the data and the purpose of the exercise?

        The dictionary is the side affect of phrases I retrieved from crawling for my applications. The purpose of the exercise is the following. I created a keyword generator which is a web application. From the logs I found that some people merge words in a seed phrase and my resulted keywords filter fails to establish similarity in these cases. So I need to split. But usually it is only a two reasonable words split so that looks like your algorithm with my dictionary works well.

        I can send you a link to the tool, but I am not sure it's a right way to do it through the forum.
[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom