I could write a long essay about why this makes very little sense, similar to the one I recently wrote about Richard Dawkins, when he made a similar error, the crux of which was below. All I would needto change is the words in bold.

The fundamental problem here is that [insert name of well-known senior figure here] doesn’t reflect on how [AI] outputs have been generated. Claude’s outputs are the product of a form of mimicry, rather than as a report of genuine internal states.Consciousness is about internal states; the mimicry, no matter how rich, proves very little. Dawkins seems to imagine that since LLMs say things people do, they must be like people, and that simply does not follow.In his framing, [Senior Figure X] confuses himself, and does violence to the concept of consciousness [and what it is to be a being]. You can’t just look at the outputs, without investigating the underlying mechanisms, and conclude that two entities with similar outputs reach those similar outputs by similar means. And the differences are immense; one (the LLM) effectively memorizes the entire internet; the other (the human) builds a mental model through experience with world.

The fundamental problem here is that [insert name of well-known senior figure here] doesn’t reflect on how [AI] outputs have been generated. Claude’s outputs are the product of a form of mimicry, rather than as a report of genuine internal states.

Consciousness is about internal states; the mimicry, no matter how rich, proves very little. Dawkins seems to imagine that since LLMs say things people do, they must be like people, and that simply does not follow.

In his framing, [Senior Figure X] confuses himself, and does violence to the concept of consciousness [and what it is to be a being]. You can’t just look at the outputs, without investigating the underlying mechanisms, and conclude that two entities with similar outputs reach those similar outputs by similar means. And the differences are immense; one (the LLM) effectively memorizes the entire internet; the other (the human) builds a mental model through experience with world.

But why bother to rewrite that essay?

A couple days ago, before Hinton’s interview, Pope Leo XIV said it better, and more concisely, in, of all things, a tweet (following up his excellent Encyclical):

True comprehension comes from experience, not text approximation.

As Valerio Capraro notes, the Pope’s point is quite similar to a point that we made together with Walter Quattrociochi in February in Nature:

LLM researchers are NOT creating beings.

They are creating interactive fiction that is trained to predict the language of actual beings.

Those two are NOT the same.

And Hinton should know better.

P.S. Want to know my thoughts on internal “emotion” states in LLMs, and why they are not compelling evidence? Stay tuned for a forthcoming essay. For bonus points, you can see my recent impromptu debate with the artist/musician Grimes on AI and consciousness, in two threads here and here.