It's all to do with the order in which operators get evaluated (precedance). In your open statement, || is evaluated before the comma, so your statement is parsed as open FILE, (">test.txt" || $failed_flag = 1); giving the syntax error.

Perl has some super-low precedance operators (and, or and not). and and or can be used as statement modifiers. So, you can write:

open FILE, ">test.txt" or $failed_flag = 1;
[download]

Or, if you add parentheses to your original statement, you can write:

open( FILE, ">test.txt" ) || $failed_flag = 1;
[download]

That said, if you're using a modern perl, you are better off using the three-argument open and a lexical filehandle:

open my $FILE, ">", "test.txt" or $failed_flag = 1;
[download]

See here to read more about precedance

update: missed a quote. Thanks toolic. not can't be used as a modifier.