Gambling may give you personal highs when you get it right, but when the overwhelming weight of evidence is that you will guess wrong.

Of course! I guess wrong all of the time. I have no illusions about never guessing wrong. And I don't try to predict the future for some "personal high". I do it because it has a proven track record of working well (because I don't let it "run away").

And the effects of those wrong guesses on you and those around you are so dire

And that's where you completely missed the point at least as far as my approach and experience is concerned. The effects of my wrong guesses are usually... none at all.

If I guess wrong (this time) about Ops (likely) wanting to be told how many calls were received in the last 5 minutes (which happens often enough, because Ops has some other way to get that information so they don't need my project to provide it for them). I lose nothing. My team loses nothing.

We didn't implement "provide the count of calls over the last 5 minutes". We didn't spend a single minute implementing even part of that feature. We implemented exactly what we needed, just in a less fragile way that didn't even add any time to our schedule (but allowed for easily implementing a lot of other things that are likely to be useful).

You present a lot of evidence that there are serious problems with spending too much time on design (and on designing too much at once). No kidding. But that doesn't prove that you can't also spend too little time on design. It takes very little experience to realize that zero design time is not a recipe for great success in programming.

I never said a single thing about spending long periods of time designing. Quite the contrary. And, by being careful to limit time spent in several ways, the risk of guessing wrong is minimal while there are often time benefits even for wrong guesses. So, on average, even if I never guess right, we win.

But I very frequently guess right. It isn't particularly hard. I know the fields I work in well. Even when I guess wrong, I'm usually still pretty close and have made good decisions about what aspects to emphasize or de-emphasize.

If simply marching across a flat field to an obvious destination is the best analogy for the programming challenges you are given, then I don't want your job. My programming challenges are much more interesting than that. That isn't a good analogy for programming. It is a great analogy for a situation where planning does absolutely no good. And if stumbling while blindfolded is the best analogy for how you implement, then I don't want to work with you. ;)

The biggest time sink in software implementation, in my experience, is time wasted going down what turn out to be dead ends. That is a big reason why "too little design" can be a serious problem when writing software. It takes planning to avoid dead ends.

And, by far, the most common reason for projects coming close to failing, in my experience, is poor initial planning leading to developers feeling "behind schedule" such that they feel they don't have time to "waste" on improving planning and organization. In an attempt to catch up or just not fall further behind schedule, they concentrate on working harder/faster, staying more "focused" on the implementation work. Their tunnel vision becomes more pronounced and they enter what can end up being a very, very long phase of constantly thinking (and reporting) that they are "almost done". Which makes them look to their bosses (and maybe to themselves) like they suck at writing software.

So I invest a fairly small amount of time up-front to accurately determine the complexity of the features being asked for and to fairly accurately nail down the landscape between the parts so that the risk of going down a dead-end is significantly reduced. And during implementation I don't just throw code at the revision control system, I track and refine a design for the component with particular emphasis to the interfaces being created and used in order to increase modularity.

You can certainly have too much design or too much organization. I've seen plenty of that. You can end up with red tape and bureaucracy. So don't do that. In almost all of the places I have worked, there is plenty of pressure to not spend lots of time on design. And at least most of my coworkers seem keenly aware of the trap of over-design and of too much "process". So it isn't hard to avoid other than the smallest excursions into over-thinking at work. It is a continuous balancing act where we avoid having way too little design and avoid more than trivial amounts of over-design.

Maybe you've worked with huge corporations or governments (or huge corporations working on government contracts)? A few times I've worked where we got absorbed by a huge corporation and I found I was increasingly impacted by an impressive build-up of bureaucracy and my efforts, even with the help of my managers, could not reduce red tape as fast as it leaked into my work area. I was quick to find new work.

I don't recall ever having been involved in a massive project, much less one that failed to be finished because the initial design was just too much. I'm sure that happened a lot in 1994. It doesn't have anything to do with what I was talking about, though.

Actually, the projects that end up being the most complicated and thus are the ones that most often get pushed off repeatedly as too risky are the projects where we are trying to rework an existing system so that it can handle new requirements. And my experience is that those situations are most common and most difficult when the developers of the system disregarded likely future concerns, the systems where they just threw code together and got it working.

Even systems where the guesses about the future were wrong, they look far enough outside their current narrow view of the problem space such that they abstract things and the abstractions at least provide places to leverage changes even if those changes are part of completely replacing the system.