Smoothing text input

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Dear Brothers,

I would like to take arbitrary text input and manipulate it a bit. The input will be in a single scalar: $string. It might be null, empty, multiline or very large. I want to break everything into "words"; trim any leading, trailing or embedded white space; and get rid of line endings.

I have tried the following but the "map" is not doing what i expected. Is there an existing module that will do this better? What am I missing in this routine? Am I on the right track?

    sub wordBreak {
        my ($string) = @_;
        my $tmp      = "";   
        my @result   = ();   
        my @array    = ();   

        #
        # break the input into "lines"
        #
        @array = split(/^/m, $string);

        #
        # turn on swallow ending white space mode and
        # chomp the array elements
        #
        $tmp = $/;
        $/ = "";
        chomp(@array) if ($#array >= 0);
        $/ = $tmp; 


        @result = split(
            ' ',    # split the one line into words
            join(
                ' ',    # join the "lines" into one
                map {
                    s/\s+/ /g;    # get rid of extra whitespace
                    s/^ //;
                    s/ $//;
                    } @array
            )
        ) if ($#array >= 0);

        return @result;
    }
[download]

Comment on Smoothing text input Download Code

Replies are listed 'Best First'.
Re: Smoothing text input by Kenosis (Priest) on Aug 29, 2012 at 16:30 UTC
You could use Text::ParseWords: `use Modern::Perl; use Text::ParseWords; say for shellwords 'This is just an average looking string.';` [download] Output: `This is just an average looking string.` [download]	[reply] [d/l] [select]
Re^2: Smoothing text input by Anonymous Monk on Aug 29, 2012 at 17:42 UTC
Thanks! I found Text::ParseWords just after the initial post	[reply]
Re^3: Smoothing text input by Kenosis (Priest) on Aug 29, 2012 at 18:14 UTC
You're most welcome! It's a good find...	[reply]
Re: Smoothing text input by johngg (Canon) on Aug 29, 2012 at 17:08 UTC
but the "map" is not doing what i expected That's because the result of the map that is passed out is the result of the last statement in it. You are not passing `$_` out of your map, just the result (number of substitutions) of the `s/ $//;`. Consider the following two code snippets. `$ perl -E ' > @arr = qw{ > abCdeFg > hIJklMn > OpqrsTu > }; > say join q{:}, @arr; > $str = join q{:}, > map { s{[A-Z]}{}g } @arr; > say $str;' abCdeFg:hIJklMn:OpqrsTu 2:3:2 $` [download] `$ perl -E ' > @arr = qw{ > abCdeFg > hIJklMn > OpqrsTu > }; > say join q{:}, @arr; > $str = join q{:}, > map { s{[A-Z]}{}g; $_ } @arr; > say $str;' abCdeFg:hIJklMn:OpqrsTu abdeg:h*kln:pqrsu $` [download] Notice the different result when I pass `$_` out of the map by mentioning it in a final statement. I hope this is helpful. Cheers, JohnGG	[reply] [d/l] [select]
Re^2: Smoothing text input by Anonymous Monk on Aug 29, 2012 at 17:43 UTC
Doh! Of course. Thank you!	[reply]
Re: Smoothing text input by choroba (Cardinal) on Aug 29, 2012 at 15:58 UTC
`$string =~ s/^\s+//; split /\s+/, $string;` [download] should work just fine. (updated). لսႽ† ᥲᥒ⚪⟊Ⴙᘓᖇ Ꮅᘓᖇ⎱ Ⴙᥲ𝇋ƙᘓᖇ	[reply] [d/l]
Re: Smoothing text input by hbm (Hermit) on Aug 29, 2012 at 19:08 UTC
Just grab the non-spaces? `sub wordBreak { my ($string) = @_; return $string =~ /\S+/g; }` [download]	[reply] [d/l]