Indeed. I'll chime in with the reassurance that there's little "wrong" with the modelling of 'physical' objects.

It tends to be a bit heavy weighted later on. But refactor it a bit at a time. For instance, everything you describe sounds ok. But I don't have to work with it, so it's tough to figure out where the sticking points are. All I can really advise is to pay very close attention to "I wish I could just...." thoughts. Optimistically write it top-down as far as you can, then the design difficulties will emerge in a pretty clear way.

Either way, sounds like your thinking on the right lines.