"Reference Timestamp" is NOT what you want.

From the RFC:

Reference Timestamp: Time when the system clock was last set or corrected

If you only want to use one timestamp (ignoring delays, etc.), use "Transmit Timestamp". That is the time at the server when it sent you it's reply.

If you want to calculate your offset it would be

0.5 * (
        ($result{"Receive Timestamp"}  - $result{"Originate Timestamp"})
      +	($result{"Transmit Timestamp"} - $time_when_reply_arrived})
      );