This looks like a job for zero-width lookahead and lookbehind assertions:
#!/usr/bin/perl -w
use strict;
my $str = "x:= y + z;";
my @tokens = split /\s+|(?<=\w)(?=(?:;|:=))/, $str;
print join "\n", @tokens;
I don't know if you find commented regular expressions helpful, but just in case you do:
my @tokens = split /
\s+ # whitespace
| # or
(?<=\w) # after a word character
(?= # before a .....
(?: # non-backreferencing paranthesis (to pre
+vent adding matches to list returned)
; # a semicolon
| # or
:= # colon equals; add more operators to spl
+it on after here
)
)
/x, $str;
