Why does Knowledge Matter?
When Wikipedia first became big on the internet, many academics and intellectuals were very suspicious of it. How could a crowd-sourced repository of information about everything on earth really be accurate? It seemed perfectly fine when you used it to find out things about which you were ignorant, but sometimes when you looked for things about your own area of expertise, it was inaccurate, eccentric, random, and messy. It was a bit like reading the newspaper. I generally trust what I read in reputable print newspapers on matters to do with politics and the world, but whenever the papers report on something I know about — philosophy or science, say — their reporting was invariably inaccurate, in small but annoying ways. (Curiously, I never generalised this lesson to the other subject-matters the newspapers reported on.)
I used to tell my philosophy students to avoid philosophy entries on Wikipedia. But things have changed. Wikipedia has improved hugely, and is still improving. The advice I give to students these days is that Wikipedia is usually a good source, though they must be careful when cutting and ‘accidentally’ pasting it into their essays. Wikipedia is now one of the many wonderful online sources available to students, many of them free and most of them well-edited.
However, the widespread adoption of generative AI has taken things in a different direction. In addition to using chatbots like ChatGPT and Claude to create documents, people use them for internet searches, to answer questions and requests for information or facts. Google’s Gemini is one of the most popular AI chatbots and is becoming integrated into the whole Google package. In a very important new development, an ‘AI summary’ is now the default first response in a Google search.
This might all seem fine — after all, don’t we all use Google and Wikipedia interchangeably without thinking, and simply rely on the results? What is the difference between looking up something on Wikipedia and using a Google AI summary? Aren’t they ultimately the same kind of thing?
I want to argue that they are actually very different, and the difference derives from fundamental facts about knowledge. Wikipedia has a right to be called a knowledge store, AI Chatbots do not. Let me explain.
In a recent interview, the Nobel Prize winner and founder of Google DeepMind, Sir Demis Hassabis said that one reason that Large Language Models (LLMs, the computational structures behind today’s AI chatbots) are so successful is that they are ‘starting with all of human knowledge, what we put on the internet.’ This claim is very revealing. On the face of it, it might seem plausible — even allowing for exaggeration — but in fact it is profoundly wrong.
Let us begin with the phenomenon of making a mistake. Everyone knows that chatbots make mistakes. Or more precisely, they convey information that isn’t true. We have all experienced it, and this tendency has not been eliminated even from the most advanced models (Google standardly reminds readers of its AI summaries that ‘all responses may include mistakes’). AI researchers have taken to calling these mistakes ‘hallucinations.’ AI hallucinations range from making up non-existent academic or legal documents, to fabricating stories about real people, to simple mathematical and logical errors.
AI scientists tend to dismiss these hallucinations as teething problems, minor malfunctions in the system which can be eliminated as the models are refined and trained further. But in fact, from the point of view of the AI machine itself — that is, the point of view of what it is designed to do — these are not malfunctions. What the chatbot is designed to do is to produce grammatical sentences in response to a prompt, whose content is relevant to the prompt. Hallucinations are therefore not malfunctions: the machine is doing what it is trained to do.
For this reason, the evocative word ‘hallucination’ is deeply misleading. A visual hallucination is when the visual system produces experiences which radically misrepresent how things really are — for example, the hallucinations of those who take LSD, or more tragically those suffering from Charles Bonnet Syndrome or alcoholism. Hallucinations are a failure, a malfunctioning in the visual system. The system is failing to do what evolution designed it to do: represent the visual world accurately. But this is not the case with the hallucinations of LLMs.
The fact that LLMs are not designed to represent things correctly also explains why it doesn’t matter, from their point of view, that not all the information they are trained on is correct. Some of it is misleading, incomplete, or plainly deceptive. This doesn’t matter if the purpose of the machine is to output grammatical sentences. But it does matter if they are meant to create knowledge.
What is wrong with Hassabis’s claim that LLMs are trained on ‘all of human knowledge,’ then, is not just that what is on the internet is incomplete, and therefore not the whole of knowledge; but more importantly, because a lot of what is on the internet is not knowledge at all. It is in the nature of knowledge — a matter of definition, if you want to put it that way — that knowledge must be true. If you claim to know that the Sun orbits the Earth, you are wrong, and therefore you cannot know it. You only think you know it, but you do not know it. And if you claim to know that MMR vaccines cause autism, you are wrong, and therefore you cannot know it, no matter how strongly you believe it.
Some people may object here that people have different ‘knowledges’ and we should not dismiss these alternative views. This is a misunderstanding. Of course, people have different and incompatible views, and how we behave towards those with different views is an important ethical and political question. But if these views are not correct, then they cannot be knowledge. Knowledge is essentially a true or correct representation of reality.
The importance of this point cannot be overstated. The concept of knowledge is central to all human cultures: not all human languages have distinct words for eating and drinking, or for he and she, but all human languages have a word for knowledge, where this implies truth. Knowledge — no matter what you call it in your language — is the concept that classifies a certain kind of correct or true representation. If it is not true, it can’t be knowledge. There is no false knowledge.
But the distinction between truth and falsehood is invisible to LLMs. They construct sentences based on interactions with billions of texts on the internet. But it is not their role — nor the role of the underpaid data-labelling employees who classify their outputs — to distinguish between true and false texts. Any text is good enough, so long as it is relevant to the prompt.
This, then, is the key to understanding the difference between Wikipedia and LLMs. The aim of Wikipedia is to get the correct information, and it tries to achieve this with its huge collection of volunteers, who are regularly updating entries and correcting the interventions of others, according to the fairly strict Wikipedia protocols. What they are aiming at is accuracy, of which truth is a specific version. Sometimes contributors make mistakes, and they are corrected by others. No doubt mistakes remain. But the point is that Wikipedia, unlike an LLM, aims to be trained on the opinions about the real world provided by the volunteers, with the overall goal of representing the world correctly. This is one of the reasons why some leading scientists who are critical of LLMs — like Gary Marcus and Yann LeCun — have argued that AI machines need ‘world models’ in addition to their text-producing abilities. How this idea will develop is one of the most interesting things in AI at the moment.
Someone may object here that most LLMs have swallowed Wikipedia anyway, so what is the real difference in principle between these two systems?
Now, of course it is good that LLMs are drawing on Wikipedia, rather than just on things like conspiracy theory websites. The difference, however, is about the aim of the systems, not just what they are trained on. Although Wikipedia contains false information, its aim is to produce true or correct information. It is a flaw in a Wikipedia entry if it contains falsehoods; but as we have seen, a hallucination in a chatbot is not a flaw, given the point of the chatbot. Wikipedia contributors are working with the aim of making the entries as correct as possible; LLM data labellers do not have that aim.
Wikipedia gives no absolute guarantee of correctness — there is no such thing. But what it gives you is a reliable way of finding things out about the world. LLMs give you a reliable way of producing coherent texts in a few seconds. But they give you no reliable way of finding out about the world. This is why Wikipedia can be a source of knowledge and LLMs are essentially not a source of knowledge, because reliability is essential to the way we acquire knowledge. Knowledge requires truth, but not just truth — because you can obtain a truth by accident (for example, by a lucky guess). What you need for knowledge are reliable methods for achieving truth — for example, by using your eyesight, appealing to established reliable authorities or by using scientific experiments to confirm theories. The idea of a reliable way of achieving truth is the essence of knowledge.
In the FWF Cluster of Excellence “Knowledge in Crisis,” we are applying these fundamental ideas from the theory of knowledge (epistemology) to all of today’s various crises of knowledge — of which the crisis of AI is only one. We have had workshops and public events discussing the relevance of the philosophy of truth and knowledge to the crises, and our researchers have published academic articles and books on these subjects. We collaborate with researchers from other fields — for example, science communication, social sciences, psychology, and economics — on questions of common interest; and this year we will collaborate with the FWF Cluster of Excellence “Bilateral AI” on public events discussing the meaning of today’s AI.
It is remarkable how deep the philosophical roots of these crises go. Sometimes the epistemological crises are discussed merely in terms of things like science denial (e.g., vaccine denial) or the problem of misinformation and disinformation, where people are being deliberately misled about political matters in social media. These are important problems, and at Knowledge in Crisis we have been working hard to understand them. But the epistemology of AI — in particular of LLMs — is something which is less obvious, much deeper and threatens the heart of our knowledge-producing practices. Unless we have a proper understanding of how the working of these machines relates to facts about truth and knowledge, we will dig ourselves deeper into the pit of ignorance and confusion which the AI companies are creating for us. One of our ambitions in Knowledge in Crisis is to create such an understanding, and distribute it to everyone.
Tim Crane, Director of Research, FWF Cluster of Excellence “Knowledge in Crisis”
About the author
Tim Crane is Director of Research of the Cluster of Excellence “Knowledge in Crisis” and professor of philosophy and prorector at Central European University in Vienna. He founded the Institute of Philosophy at the University of London and was Knightbridge Professor at the University of Cambridge.