You are adding memory addresses.

If you change what $b points to, the add it to $a, the $sum will change.

You state : I make a change to hashref $b

Did you change the internal content of the hashref, or did you assign a new hashref to $b ? (Thereby changing the memory address)