Why AI doesn’t always get it right

You’ve probably heard about some of the times when AI doesn’t always get things right. Whether it hallucinates a citation, gives you a bogus recipe, or mixes up a fact, you know that AI isn’t always correct. But do you know why?

Large Language Models create human-like language using statistics. They are programmed, based on large amounts of data, to create responses based on the statistical likelihood words come next. They learn, although not in the same way humans do, patterns from the data they are trained on. First, they utilize patterns they have learned related to the prompt you input. Then, they pick each word based on the statistical likelihood, garnered from their data, that it will come next.

This process is why sometimes AI hallucinates. Based on statistical likelihood, AI makes a guess at what should come next. And sometimes that guess is wrong. While it can reference facts and correct resources, it is never guaranteed to be correct. There are other reasons AI can be incorrect too. Because it is trained primarily on internet sources, sometimes incorrect information that is listed online comes through. This is especially challenging to identify when AI produces answers without explaining where its answers came from. Additionally, AI is trained on data that may include biases and lack a variety of valuable perspectives. This is because a lot of the data comes from specific places and often reflects a singular world view.

Risk and Weaknesses

Utilizing Large Language Models when searching for factual information is always a risk. By becoming aware of the risks, you can use AI more effectively.

Large Language Models are not a high-quality source of information.

- You may need to fact check all or part of what you find because Large Language Models sources from a wide variety of resources, including those that are not factually correct.

Hallucinations may cause problems.

- You may need to fact check all or part of what you find because Large Language Models can hallucinate based on the way they generate human-like language.

Large Language Models have privacy concerns.

- Some Large Language Models are trained on data that were not ethically or legally obtained. Further, public models, unlike those provided by IU, are trained on data that you put into them. This is also an unsettled area of copyright law.

The answers you receive are biased based on the data entered into the system.

- When interacting with LLMs, it’s important to remember that the information it provides you with is generated based on the data it was trained with, which it is subject to bias. Therefore, the information an LLM outputs may also be biased. For example, these data sets may only reflect views and opinions from one part of the world, lacking a variety of global perspectives and realities.

References:

Bergmann, D. (2025, August 18). What is machine learning? | IBM. https://www.ibm.com/think/topics/machine-learningLinks to an external site.

Denning, P. J. (2025). Artificial intelligence: Generative ai. Ubiquity, 2025(July), 1–19. https://doi.org/10.1145/3747356