I had to do something very similar. I started as a contractor and noticed that when I was hired that my hours came under serious scrutiny (primarily because I come in around 9:30 am to 5:00 pm, and then from 9:00 pm to around 2:00 am). I too started tracking my time and the very next time that someone fired off a snide remark I went back to my desk, printed off a week's worth of timesheets and then promptly presented the crow to be eaten sans fork ;-)

I haven't had anymore trouble about the hours that I work