I'm supposed to wear something to work? I wondered why everybody looked at me funny.

Seriously, sometimes I am in jeans, sometimes I wear slacks and a tie. As bastard pointed out, quality is everything. If you expect to pay the same for a dress shirt and slacks as you pay for a jeans and a t-shirt, you're darned right they're not going to be as comfortable! If I have a nice button-down shirt, with a tie and a nice pair of slacks, not only am I comfortable, I look better, too (I need all the help I can get, as my monk pic in the upper right demonstrates).

In fact (I'm almost embarrassed to admit this), before I met my last girlfriend, I didn't own a single pair of jeans and I only had one short-sleeved shirt (which was a gift from another girlfriend). I've loosened up a bit.

Mostly, though, I think feeling comfortable is a matter of attitude. If you're comfortable with your clothes, you'll be comfortable in your clothes.

Cheers,
Ovid