Language models don't "know" anything
Training data are construction materials for a language models. A language model can never be inspired. It is itself a cultural artefact derived from other cultural artefacts.
The machine learning process is loosely based on decades-old grossly simplified models of how brains work.
[…]
It’s important to remember this so that we don’t fall for marketing claims that constantly imply that these tools are fully functioning assistants.
[…]
Who was the first man on the moon?
[…]
What you’re likely to get back from that prompt would be something like:
On July 20, 1969, Neil Armstrong became the first human to step on the moon.
This is NASA’s own phrasing. Most answers on the web are likely to be variations on this, so the answer from a language model is likely to be so too.
[…]
The prompt we provided is strongly associated in the training data set with other sentences that are all variations of NASA’s phrasing of the answer. The model won’t answer with just “Neil Armstrong” because it isn’t actually answering the question, it’s responding with the text that correlates with the question. It doesn’t “know” anything.