The square brackets ([]) delimit a character class, which essentially means "match any of these characters". So in your example, the regex ([^to ]|[^from ]) would match any character that is not (because of the ^ character which negates the character class) one of the characters f,m,o,r,t or a space.

You can use simple alternation to match either of these words:

if $line !~ /train times (?:to|from)\b/ {
    # do something with line
}
[download]

Update: added word boundary \b, thanks to GrandFather for pointing this out :-)