I was expecting this, self-referential problems are the core of many proofs like Gödel's incompleteness or Turing's halting problem.

Your author just wrapped a story around it. :)

Which is also a reason why you can't generally prove that algorithms are correct.

To get back to the topic, the problem with AI/LLM at the moment is that they are based on statistical pattern matching not logic.

Simplified, they might produce solutions on assumptions like "all odd numbers are prime" because they saw the pattern till 7.

(I hope there is a God protecting us once a vital system encounters 9. ;)

This is simplified because LLMs are normally trained on human input making clear that 9 isn't prime.

But the lack of more human input - the Internet is finite - is the sharpest needle pointing at the current bubble.

Still AI is very good at areas which don't require logic, like English orthography, and helping me finding the distinction between "proofs" and "proves" ;)