1 Building with GenAI

How do AI systems communicate in fluent, human-like language? Does ChatGPT truly think like we do, or is something else at play? In this book, we’ll explore what makes GenAI so unique, mysterious, and fascinating.

Our goal is to help you understand how GenAI applications work. We’ll consider what tasks they can handle well and where you should steer clear. Along the way, we’ll roll up our sleeves and build a few applications of our own using a low-code tool called LangFlow. To begin our journey, let’s take a look at the fundamental characteristics that set GenAI applications apart and make our ride worth the effort.

In traditional programming, you instruct a machine to perform certain steps in a specific order, forming a logical workflow that transforms your input into the desired output. These steps can be described through code, as the application’s behavior will be mainly deterministic. Such codification happens by means of a formal language whose specific syntax and semantics define the particular programming language, such as Python or C, the programmer will use.

In GenAI programming, you have an additional element in the picture, the Large Language Model, or LLM. At some point in your workflow you will pass through this “magic black box” that takes text as input and generates text as output. As you design the workflow, you will have to take care of how to craft the right text that goes as an input to the LLM. Such a text is not written in a programming language but in natural languages, such as English or Italian, which humans naturally use to communicate. Being that the LLM is a black box, it does not matter how carefully you craft your prompt. You cannot be 100% sure of what exact output you will get in return. The magic black box brings inherent randomness into the GenAI application’s behavior.

Of course, one can say that what we call traditional programming might also include some of these elements of strangeness. For example, in AI applications (not specifically of a generative nature), we also must come to terms with the wild behavior of probabilistic models. However, in GenAI, we will always have to deal with these elements at once: probabilistic behavior and natural language instructions. So, we better get used to the idea that we need to get very good at managing them effectively, as we will learn to do throughout this book.

These elements are unmissable in GenAI because of the component that turns a generic application into a GenAI application: the LLM. We will give it a shot to understand how LLM works later. For now, it is enough to imagine an LLM as a seemingly magic black box that takes some text as input and generates some other text as an output.

For example, the iconic 2022 ChatGPT by OpenAI does not seem to be affected by such restrictions, although it is clearly based on LLMs. ChatGPT appears to have some memory: it remembers what we said in the earlier part of a conversation and builds upon it. It doesn’t look quite stateless, really. The OpenAI engineers working on ChatGPT clearly managed to “tame the beast” and overcome the memoryless limitation.

Also, GenAI applications used in the workplace, like 2023 Microsoft CoPilot, seem to learn from your documents and provide answers to specific knowledge that wouldn’t have been available originally when the underlying LLM was built. In other words, their understanding of the world doesn’t seem limited to what they have been once pre-trained on, but it seems to be expandable.

Let’s see how this works through an example.

For example:

Let’s have a look at this example:

Still, you might ask yourself: how can the application equip the input for the LLM with the proper knowledge? Well, we will learn what techniques we have available for doing so in the following chapters. For now, think about two simplistic but fair options: one is to always attach the full knowledge to each query. If the knowledge is not very large, you can afford to do so in every call to the LLM. Another option would have been to run a simple, deterministic search query on a document repository, find all the sentences that include relevant keywords (like “office” in this case), and only attach those. We’ll come back to this—for now, you got the gist.

Just as we don't need to know how an engine works to drive a car, we don't need to delve into the intricate workings of an LLM to build generative AI applications. In the following pages, you will learn how large language models work. We will break down complex concepts into simple, digestible explanations.

Fortunately, you can extend this output to complete entire phrases and paragraphs, adding more than one word. In fact, if you run the completion step repeatedly, you will get an additional word each time. Such a model is called autoregressive because it is applied over and over, using as an input the previous iteration's output to build more complex completions than just one word, creating complete sentences, paragraphs, poems, songs, or entire reports.

Let's build an intuition for how GPT works by acquainting ourselves with its way of representing text. In the world of GPT, phrases are "tokenized" which means they are split into units called tokens. We can think of a token as a word, a part of a word (some more complex English words are represented as a composition of multiple tokens), or a punctuation sign. These tokens are then translated into numbers through a process called embedding.

Let’s build a simplified example and imagine a basic GPT that uses a dictionary of just eight words: "THE, CAPITAL, OF, ITALY, IS, PARIS, ROME, LONDON". Each word in the dictionary is associated with three numbers (its embeddings) and can be visualized in a three-dimensional space, as the figure below displays. These tuples of numbers, the embeddings, will be the atomic units used for computations.

What does the GPT algorithm do? First, it needs to build a context starting from the prompt, using some calculation based on the embedding coordinates. For instance, we can imagine that by calculating the arithmetic mean, coordinate by coordinate, of the words constituting the prompt, we obtain a new tuple of numbers that we will call the context.

As shown here, the x-coordinate of the context is computed by averaging out the x-coordinates of the words composing the prompt. This is, of course, a simplification of the actual calculation process happening within GPT, but let’s use it for now to help us build an intuition about how this seemingly magical process works.

See the example below: the coordinates of the candidate words are combined with the coordinates of the context using a combination—technically, it would be called an inner product—of the two sets of numbers representing each candidate word and the context. The word obtaining the highest result wins. In this case, “The capital of Italy is” obtains “Rome” as its best completion, as the resulting probability for “Rome”, 0.47, is higher than what the other candidate words get. Well done to our simplified GPT for having guessed the right completion!

The simplified process we just saw employed a bunch of elementary calculations, which could, in theory, be done with pen and paper. Linear combinations use only multiplications and sums, the dictionary of only eight tokens, and the embeddings of three numbers for each token give us 24 numbers to start from for doing completions: it’s too simple to be true.

In fact, the reality of modern LLMs and, specifically, GPT-3, which is the one used within 2022’s ChatGPT, is much more complex. Let's delve into the complex reality of GPT-3 by examining three key aspects of the simplified methodology we discussed earlier: the dictionary of embeddings, the calculation of the context, and the calculation of the next best word.

You’ll encounter many new concepts and technical terms in the next few pages. But don’t worry—you don’t need to become an expert on every detail of modern LLM architectures. Think of this as a window into what’s happening under the hood: it’s here to give you a taste of how these advanced tools actually function. If you find some parts too dense, feel free to skim or even skip ahead to the final section, “Keeping the Knowledge Up to Date,” without missing the bigger picture.

You can clearly see how the calculation of sending the ordered set of tokens of the prompt through deep neural networks with billions of parameters is a much more complicated way to obtain the context than a simple average of the same tokens would do. This is what it takes to build an LLM that seems confident about what it is talking about, whatever the subject.

In summary, the figure below shows how a GPT architecture can be broken down into three main blocks. First, incoming words or tokens are converted into embeddings—large sets of numbers that capture their meaning. Next, these embeddings pass through deep neural networks (the Context Calculation and the Combination steps), which use attention and positional information to produce a probability distribution over all possible next tokens. Finally, the winning token—the one with the highest probability—is selected and added to the prompt. That expanded prompt then re-enters the cycle, generating new embeddings, an updated probability distribution, and so on.

In unveiling step-by-step what happens under the hood with GPT-3, we are moving backward and have started from the second stage. Across the steps we have seen so far (translating words in numbers, calculating context, and combining it with the prompt), a given set of parameters is applied to an input (the prompt) by means of some complex computations. But where do such given parameters used during inference come from? In other words, how do we obtain the precise set of numbers that produce “Rome” as the completion of our sentence on the capital of Italy?

This is what happens during the first stage of an LLM lifecycle, the model training. The ‘P’ of GPT stands for pre-trained because the parameters able to make such a marvel happen during inference are found once and for all during the preliminary training process. Let’s see how this stage works.

To start, we should build an intuition of how neural network models are trained. Neural networks are built through a process that starts with random parameters, which are numbers that influence how the network makes decisions. Think of it like a student starting with no knowledge and guessing answers. Initially, these guesses are often wrong, but with each new piece of information (or training example), the student adjusts their guesses to get closer to the correct answer.

This adjustment happens through a series of iterative learning passes, often described as "backward" learning or backpropagation. Here's how it works: the neural network makes a prediction based on its current parameters. If the prediction is wrong, the network goes back and tweaks the parameters slightly to improve future predictions. This process repeats many times, with the network constantly adjusting its parameters to reduce errors.

Training requires a lot of human-intelligible text. Each time the network sees a new piece of text, it learns to better predict and complete sentences. To train effectively, the text must cover diverse subjects, from different disciplines. This helps the network learn to handle a wide variety of contexts and topics.

For GPT-3, the OpenAI team used a massive collection of text, amounting to the gargantuan size of 45 TB of textual archive. Such a learning dataset included text from various sources, such as websites (using pages “scraped” from blogs, forums, news portals, and so on), books, the whole of English Wikipedia, and academic articles, providing a rich and varied dataset for training. By processing this extensive and diverse text, the neural network parameters gradually converged to values that enable accurate and contextually appropriate text completions.

Conversely, the inference process is dynamic; it occurs in real time based on the user's prompt. Before calling the LLM, the prompt must be enriched with any additional or updated information, leveraging specific prompting techniques, as we will learn in the book. Essentially, we need to equip the LLM with additional mechanisms, devices, and customization to be able to interact with all the knowledge it needs, integrating its pre-trained knowledge with whatever else it needs.

Another important consideration is the concept of fine-tuning, which allows us to enhance or specialize a pre-trained model. Fine-tuning involves taking a pre-trained neural network like our GPT-3 and conducting additional learning passes with new, specific, or more relevant data. This process can yield multiple benefits: it can make the LLM more adept at specific tasks by tailoring it to particular sets of instructions, or it can integrate specialized knowledge from domains that were underrepresented in the initial training data, such as a specific branch of medicine. Additionally, fine-tuning can improve the model's ability to maintain dialogues with humans, as demonstrated by ChatGPT, which fine-tuned GPT-3 for better conversational capabilities. This fine-tuning process follows a typical supervised learning approach. The model is provided with numerous prompts and the desired completions, so that it can learn by those given examples. The neural network adjusts its parameters to converge towards the desired completions through backpropagation-like learning passes, enhancing its performance and applicability.