GenAI - the AI revolution

In the first part of our series, we learned about the different types of AI and learned that generative AI is "only" a subset of AI. This is not to belittle it. There are two aspects that need to be emphasized:

• The universal use: Before generative AI, I had to collect data on a topic I wanted to work on with AI and train a model. Only then could I use the model to solve a topic-specific task, such as credit card fraud detection. I had to do this for every topic I wanted to work on. With generative AI, I now have a large model that can perform a wide variety of tasks.
• Democratization: Whereas I used to need specialists to collect the right data and train the model, I now have a large language model that anyone can use.

How language models work

But when it comes to using them, it turns out that some of us are better at it than others. This has to do with the ability to prompt, but also with understanding how a language model works in the first place.

The mathematician Dr. Helmut Linde published an article on golem.de about how language models work and wrote this sentence: “How should I know what I am thinking before I read what I am writing?” He gets to the heart of what language models do. They just find the next word for a given text and repeat this process until there is enough text. This means that when we ask a language model a question, it determines the first word of the answer, reads the question and the first word of the answer, determines the second word of the answer, and so on.

What language models are not and cannot do

Why is this important to know? It teaches us what language models are not and cannot do:

• They are not thinking entities
• They are not search engines

It also teaches us what language models cannot do:

• They cannot check facts
• They cannot check sources
• And most importantly, they cannot cite their sources

Checking facts and cross-checking sources can be taught to a language model in an application, so to speak. This is why chatbots were very quickly equipped with Internet search. It also made up for the fact that language models are always based on an outdated dataset when they become available in the form of chatbots. However, language models will not learn to specify their own sources. At least not if they are trained the way they are today.

Understanding how language models work will make it much easier to recognize their limitations and where the risk of hallucination is greatest. That is what the next part of this series will be about.