> "There is hardly anything that people would not do in order not to have to think."

I really like that quote, but couldn't find the source. (Just a similar one from Jung and avoiding ones soul.)

The text was / is German, it was something close to "Es gibt nichts, was der Mensch nicht tun würde, um nicht denken zu müssen." (And I don't know the source.)

Regarding your analysis, it's sound if you only expect LLMs to continue the same way, that is being trained on human data stolen from the Internet.

Because that's what gains the most attention in the media and in the public, and that is what is commonly called "AI", at least in Germany

But do you remember this Go playing program which trained itself and was beating all human champions? (See AlphaGo Zero)

Faintly.

I could imagine future helper AIs interacting with software designers who provides kind of a test suite defining the expected outcome and a self trained machine provides solutions.

But this software designer will need a lot of skills, more than the usual user of ChatGPT.

Because like always this code has to stay maintainable.

I've recently watched a public talk at DESY in Hamburg, where one of the presenters talked about training neuronal networks to enhance medical diagnostics. DESY plans a new x-ray facility (PETRA IV), and he proposed to analyze medical samples both using conventional medical imaging (CT, PET, MRI, ...), and using the new x-ray facility. Of course, you can not use the high-energy x-ray facility on living tissue, it would be lethal. But the image quality is way better than what you can get from conventional medical imagers. The idea is to train that neuronal network on all images of those samples, and then use that neuronal network to help analyze the results from the imaging machines in hospitals. Effectively, the neuronal network would become an expert at interpreting imaging results with a lot of experience gained from the high energy, high resolution PETRA IV images.

Compared to PETRA IV, those medical imaging machines are dirt cheap (sorry!), and a software update and perhaps a more powerful PC connected to the imaging machine would be sufficient to improve the diagnostics.

Of course, this is digging down in the noise floor of the medical imagers. A neuronal network may find patterns that a human can't see, but it can't work miracles.

This is, like the Go problem, a very limited problem, compared to the generic "fake a 24th century computer" problem.

And, to continue comparing with dogs, that would NOT be a talking dog. It would be a guide dog, or a detection dog, or some other working dog. They can do impressive work, in their field. And we trust them, in their field. You would not expect a drug detection dog to lead a blind human, or vice versa.

In contrast Vibe programs from amateurs are likely to be throw away code.

I don't think so. Yes, people may start with throw-away code. But again, people are lazy: "Hey, $AI can generate helloworld.c and trivialsort.c. Let's make it write a solution for $complexproblem." I've seen exactly that lazyness in code cobbled together from ten or twenty first results from Google and Stackoverflow.

In the end all of these are economic questions

• how expensive will it be to keep the code maintained?
• (Update: Or can we just keep and modify the prompts to always create new programs instead of maintaining them? )
• Or even more basic: Can we afford/risk to rely on computer systems which are not anymore maintainable by humans?

Let me ask an evil question: How much code is currently maintained that should be maintained? There is tons of unmaintained and probably unmaintainable code hidden below wrappers and abstraction layers. "AI" will make that only worse.

One of my current projects at work suffers from exactly this problem. We outsourced an isolated part of the problem, and got back code that is in dire need of reworking or rewriting. It is far away from what we specified, both formally and in requirements. We lack both time and money to fix that, and so we currently add layers of toilet paper over that poop to hide the smell. (To be fair, our written spec lacked a lot of details due to a lack of time, but what we got back did not even match that spec. It is almost a not even wrong problem.)

We are not the first to do so, and "AI" won't improve the situation. Probably, we will in future feed "AI"s with junk code and ask them to fix the current problem. Then start the compiler, perhaps run a few minimal tests, and release. Guess what will happens when someone gets the task to write test cases and has access to an "AI".

Clients are rarely willing to pay for code quality. The medical and aerospace industries have come to the conclusion that a minimum of risk management, project management, and software design is required, but people had to suffer or die for the industry to learn that lession, over and over again. I think the automotive industry is on almost the same level. And still, people still desperately look for ways to bypass risk management, project management and software design to pay less and get results faster.

Alexander