When you ask ChatGPT to write a Python script, summarize a 50-page PDF, or write a poem in the style of Shakespeare, it responds in seconds with fluent, coherent, human-like text. It feels like there is a tiny conscious mind sitting inside a server farm, thinking about your question. But what is GPT actually doing under the hood? Is it really "thinking", or is something else happening?
01. The Illusion of Intelligence: Next-Token Prediction
At its fundamental core, GPT (Generative Pre-trained Transformer) does only one single thing over and over again:
Given a sequence of words, what is the most statistically probable next word?
That's it. GPT is not a database searching through files, nor is it a logical reasoning engine executing code in a sandbox. It is the world's most sophisticated, high-dimensional autocomplete engine.
When you type "The sun rises in the...", GPT doesn't look up an astronomy book. It calculates probabilities based on trillions of text patterns it saw during training and calculates that "east" has a 99.4% probability of coming next. When it picks "east", it appends it to the prompt and asks itself again: "The sun rises in the east..." — what comes next?
02. Step 1: Tokenization — Chopping Text into Pieces
Neural networks don't understand letters or words directly — they only understand numbers. Before GPT can process your prompt, it passes through a Tokenizer.
Tokenization splits text into small chunks called tokens. A token can be a full word (like cat), a subword (like ing), or even a single character. On average, 100 English words equal roughly 130 tokens.
- The sentence:
"Prompt engineering is awesome!" - Tokenized:
["Prompt", " engineer", "ing", " is", " awesome", "!"] - Numerical IDs:
[34521, 14209, 292, 318, 12490, 0]
This is why API pricing is based on tokens rather than words, and why LLMs sometimes struggle with character-level puzzles like "How many R's are in the word 'Strawberry'?" (because the model sees the token Strawberry as a single number ID, not individual letters).
03. Step 2: Vector Embeddings — Turning Words into Math
Once text is converted into token IDs, GPT maps each token into a High-Dimensional Vector Embedding Space (often 4,096 to 12,288 dimensions!).
Think of an embedding space as a massive multi-dimensional map where words with similar meanings are grouped close together:
"king"and"queen"live in the same neighborhood."apple"and"banana"sit near each other in the fruit sector."JavaScript"and"Python"cluster in the programming language sector.
Because words are represented as geometric vectors, the model can literally perform vector math on concepts! The classic example:
Vector("King") - Vector("Man") + Vector("Woman") ≈ Vector("Queen")
04. Step 3: The Secret Sauce — Self-Attention
Older AI models processed text one word at a time, forgetting the beginning of a long sentence by the time they reached the end. The breakthrough in 2017 was the Transformer architecture, created by Google researchers in the paper "Attention Is All You Need".
The key superpower of Transformers is Self-Attention. Self-Attention allows every token in a prompt to look at every other token simultaneously and decide how much "attention" to pay to it.
Consider these two sentences:
- Sentence A: "The river bank was muddy."
- Sentence B: "He deposited money in the bank."
How does GPT know that "bank" means land in Sentence A, but a financial institution in Sentence B?
During Self-Attention, the token bank in Sentence A looks at river and muddy, shifting its mathematical embedding toward geography. In Sentence B, bank looks at money and deposited, shifting its embedding toward finance.
05. Step 4: Pre-Training vs RLHF Fine-Tuning
A raw GPT model right after pre-training on the internet is actually terrible at answering questions. If you type "How do I bake a chocolate cake?", a raw completion model might complete it as: "How do I bake a chocolate cake? Here are 10 other baking questions people ask." (because it seen internet forum pages).
To turn a raw completion model into a helpful chatbot, AI labs use a two-step process:
- 1. Pre-Training (Unsupervised Learning): The model reads billions of web pages, books, and code repositories to learn grammar, facts, coding patterns, and reasoning. This costs tens of millions of dollars in GPU compute.
- 2. RLHF (Reinforcement Learning from Human Feedback): Human annotators rank model responses, teaching the model to follow instructions, be polite, decline harmful requests, and act like an assistant.
06. Why LLMs Hallucinate (And Why Confidence ≠ Truth)
Understanding what GPT actually does explains why AI hallucinates (invents fake facts or non-existent package libraries).
Because GPT predicts text based on statistical plausibility rather than a database lookup, its goal is to generate text that looks plausible. If you ask about a niche topic where training data was scarce, GPT will generate names, citations, or functions that match the statistical pattern of real citations — even if they don't exist!
To an LLM, a confident wrong answer and a confident correct answer use the exact same mathematical mechanism.
07. Summary & Mindset for Developers
Key Takeaway: Remember that GPT is a probabilistic pattern matcher, not a database. When writing code with AI, always verify logic, check package imports, and provide explicit context. When you master prompt engineering, you are giving the model the exact context vector it needs to predict the correct answer!
Understanding the machine behind the curtain is what separates developers who get confused by AI from developers who harness it to build incredible software.