Personally I think the comfort depends on the quality of the clothing. With incremnetally more leeway being granted as the clothing gets more casual. (but not all t-shirts are made equal either, its just harder to screw them up)

For example I am at home (just got here from work), and wearing trousers and a button down collar dress shirt (Geoffery Beene rules!). I probably will wear aforementioned attire nearly until I go to sleep. Quite comfortable. I think the problem arises when someone goes out and just buys the first dress shirt and trousers they see (spending as little as possible) and are duly unsatisifed with the fit/comfort level. Most of what i've tried i hate and would never wear. What I am wearing now is just fine tho.
(For reference I wear Geoffery Beene shirts and Slates trousers.)

BTW- my tye wearing thing was ended when i was told to to come into work the next day not wearing one (perhaps i'm ill, but sometimes i like them)