Well, you can build up a regexp that recognizes all of your tokens... For example,

  $word = '\w+';
  $path = '(?:/$word)+';
[download]

and so forth, then have

  $pattern = '(?:$path|$word|$number|$operator)';
[download]

or some such. To grab all the tokens off your data string, you could just do

  my @tokenList = ();

  while($data =~ /($pattern)/g) { push (@tokenList, $1); }
[download]

which will parse the string left-to-right (maintaining the order you require) pulling off individual tokens and storing them in an array. You don't even need to worry about the whitespace, because the while(//g) {} will ignore whitespace (if it's not part of your token pattern) and just grab off the tokens...

Hope this helps,
CJW