I got bit by precedence today, and as it turned out the nature of my data complicated my debugging efforts. In the end nothing I discovered was all that surprising, but the road to get there seemed dark and twisty enough to merit a 'lesson learned' posting.

I had the following data: 

%dest = ();

%source = ( very_long_key_name => 1,
            another_key => 2 );

%names = ( 1 => 'name1', 2 => 'name2' );

[download]

I simply wanted to copy $source{very_long_key_name} to the %dest hash, doing a lookup in %names along the way. Specifically, I wanted to do this: 

$dest{name} = $names{ $source{very_long_key_name} };

[download]

I ran the code and (thanks to warnings) got several 'Use of uninitialized value' warnings. A few strategically-placed print statements indicated that some of the data is missing. No problem - I'll just check for that before doing the lookup in %names. 

$dest{name} = exists $source{very_long_key_name} ?
              $names{ $source{very_long_key_name} } :
              '';

[download]

My caffeine-starved brain thought the ''; looked lonely on a line all by itself, so I reversed the condition and moved it up. (In retrospect this was excessively silly - I said I was in need of caffeine.) 

$dest{name} = not exists $source{very_long_key_name} ? '' :
              $names{ $source{very_long_key_name} };

[download]

But when I ran it, $dest{name} = 1 instead of 'name1': 

print "name code = $source{very_long_key_name}\n";            # 1
print "converted = $names{ $source{very_long_key_name} }\n";  # name1
print "dest name = $dest{name}\n";                            # 1

[download]

In reality, the data structures were much more complex and I thought I screwed up one of the dereferencing steps. I tested each one and everything seemed to be working fine in isolation. I could not figure out why the original value in $source{very_long_key_name} was getting copied rather than the lookup value in %names, until it dawned on me that the source of the '1' might not be what I thought it was.

Sure enough - a quick glance at perlop confirmed my suspicions: the conditional operator has higher precedence than 'not', so the code was being parsed like this (output from B::Deparse): 

$dest{'name'} = (! (exists($source{'very_long_key_name'}) ? '' :
                   $names{$source{'very_long_key_name'}}        ) );

[download]

Since exists($source{'very_long_key_name'} evaluates to true, the right hand side simplifies to not ''. The empty string evaluates to false, so not '' is true, or 1. (FWIW, not 'scalar' = '' and not undef = 1.)

A set of parentheses did the trick: 

$dest{name} = (not exists $source{very_long_key_name}) ? '' :
              $names{ $source{very_long_key_name} };

print $dest{name}; # name1

[download]

That's my 'duh' moment for the day. Nothing fancy, nothing complicated, and nothing that couldn't have been prevented by simply remembering precedence.

Replies are listed 'Best First'.
Re: not 'another victim of precedence' ? 'It is true' : 'the code is false'
by davido (Cardinal) on Apr 06, 2005 at 06:20 UTC

    It may be helpful to keep in mind that anytime a logical operator has both a symbol form, and a word form, the word form is always designed to be very very low precedence.

    Examples:

    • and is much lower precedence than &&
    • or is much lower precedence than ||
    • As you found, not is much lower precedence than !

    I mention this because hopefully this consistant behavior will be easier to remember once its been pointed out explicitly.


    Dave

[reply]
[d/l]
[select]

      Thanks - that's a very good point and I'm glad you mentioned it. I know the word forms are lower precedence than the symbol forms, * trumps +, and four of a kind beats a full house, but I still need to consult the precedence table regularly to keep them all straight. :)

      If in doubt (or to clarify the code) I'll usually either split the statement up (as friedo mentioned) or just add a set of parentheses. This time, however, I jumped to an incorrect conclusion regarding the source of the problem, and precedence didn't cross my mind until later.

[reply]
[reply]
      anytime a logical operator has both a symbol form, and a word form, the word form is always designed to be very very low precedence

      Indeed. I like to think of the spelled-out logical ops as almost being a special class of control-flow operators, useful for things like this:

      dosomething with some . very(comple)x expression or warn "Some Error Message";

      The spelled-out logical ops are more verbose (in terms of the number of characters) than the punctuative ones, and they connect more verbose things -- whole expressions. If you need to perform a logical operation on a single term, then you either want to use the punctuative logical ops, or parenthesize. (I tend to do the latter, because I like the way (foo and bar) reads better than foo && bar, but I'm pretty sure that's a personal idiosyncracy I have, and the other way should work just as well.)

      "In adjectives, with the addition of inflectional endings, a changeable long vowel (Qamets or Tsere) in an open, propretonic syllable will reduce to Vocal Shewa. This type of change occurs when the open, pretonic syllable of the masculine singular adjective becomes propretonic with the addition of inflectional endings."  — Pratico & Van Pelt, BBHG, p68
[reply]
[d/l]
[select]
[reply]
[d/l]
[select]

        Should I be proud that I seem to have forked Perl?

        =cut
        --Brent Dax
        There is no sig.

[reply]
      It may be helpful to keep in mind that anytime a logical operator has both a symbol form, and a word form, the word form is always designed to be very very low precedence.

      Indeed. It occurred to me that I've never been caught by this because I never use the not operator, always using !.

      That made me wonder why Perl has not. I make use of the distinction between or and ||, but under what circumstances would not be useful?

      Smylers

[reply]
Re: not 'another victim of precedence' ? 'It is true' : 'the code is false'
by friedo (Prior) on Apr 06, 2005 at 06:17 UTC
    An excellent lesson indeed. I still get tripped up with precedence issues all the time, despite my years of experience, which is why I tend to split complex operations into several statements if I can. (Though I do have a bad habit of writing crazily nested map/sort/grep blocks. But that's just 'cause it's so damn fun. :) )
[reply]
Re: not 'another victim of precedence' ? 'It is true' : 'the code is false'
by bluto (Curate) on Apr 06, 2005 at 15:32 UTC
    That's my 'duh' moment for the day. Nothing fancy, nothing complicated, and nothing that couldn't have been prevented by simply remembering precedence.

    And if your memory is faulty like mine, and you can't remember a zillion precedence rules for various languages if your life depended on it, adding extra parens is the key. It sure does seem like wimping out though...

[reply]

      I'm not sure sure it's wimping out. If I write as though the code needs to be read by someone else, I always end up using an order that makes sense when read aloud and then using parentheses to make it actually work. If I can't easily decompose the statement then I end up scratching my head a lot when I have to twiddle it six months later.

[reply]
      And if your memory is faulty like mine, and you can't remember a zillion precedence rules for various languages if your life depended on it, adding extra parens is the key.

      I disagree. In fact, I disagree that you should even try to learn the precedence rules (for Perl, at least). Larry et al put effort into ordering the relative precedence of the operators so that most of the time they will Do The Right Thing, without you having to think about precedence or parens.

      Your claim of having a "faulty" memory suggests you think that 'better' programmers have memorized all the rules. If knowing the rules was the only sane way of using the operators then precedence could be entirely arbitrary (or alphabetical, or non-existent, with everything at one level and being processed from left to right); it doesn't matter what the order is, cos it's just one of those things everybody has to learn by rote.

      But the whole point of precedence is to make life easier for programmers, so that operators naturally behave themselves. Treating precedence as some random list to be memorized is shunning this help and making life harder for yourself.

      Arguably what tripped up the original poster in this situation was using the wrong operator: not, rather than !. Knowing the operators and what they are intended to be used for is definitely worthwhile.

      I honestly didn't know the relative precedence of the operators mentioned in this thread, but I'd've happily coded using them, trusting Larry to have got them right. And ditto for associativity.

      It sure does seem like wimping out though...

      And adding extra parens can often make code harder to read. If I'm skimming through some code at speed trying to get an overview of what it does and I see a long expression without any parens then the expression probably isn't doing anything out of the ordinary and does what I expect it to do without spending much time thinking about it.

      But if I encounter a long expression with parens in it then the parens are overriding the normal precedence rules: the statement is doing something funky, so I'd better pause and read it. Moreover, working out exactly what involves matching up all the pairs of parens.

      Now it may turn out that the statement does exactly the same as if it hadn't been liberally sprinkled with parens, but it's impossible to tell that at a glance: two statements that differ just in the location of one paren may do very different things but look almost the same to a passing human eye, whereas a statement that has no parens in it is clearly and readily distinguishable from either of them.

      Trust Larry more! Use parens less!

      Smylers

[reply]
        Your claim of having a "faulty" memory suggests you think that 'better' programmers have memorized all the rules.

        What my claim suggests is that I'm an old geezer. You are reading too much into my semi-serious comment -- I should have put some smileys in there. :-)

        Really, I've been programming for about 30 years now, over 20 in C, and in the early days IIRC C compilers weren't quite standardized on precedence. Even in modern times and in perl, I found out the hard way (as the original OP of this thread) that precedence works the way you think it should 99% of the time, but there are times when it will bite you. Those times are almost not worth risking since they can be very subtle to debug -- and you feel like a total idiot when you find your mistake. FWIW, In reality I use as few parens as I can get away with and not start worrying.

        Trust Larry more! Use parens less!

        I do, trust me -- I don't whip them out like a crazed lisp programmer. A few more parens isn't going to cause a meltdown.

[reply]
Re: not 'another victim of precedence' ? 'It is true' : 'the code is false'
by periapt (Hermit) on Apr 11, 2005 at 12:01 UTC
    That very think has burned me a couple of times also. That is why I surround the condition with parenthesis by default these days. Yeah, most of the time it isn't necessary but it doesn't hurt and the one or two times it is necessary, it has saved a lot of what is left of my thinning hair.

    Nice writeup

    PJ
    use strict; use warnings; use diagnostics;
[reply]

Back to Meditations