How GPT-2 Breaks Down Into Embedding, Attention and the MLP Layer

A new explainer website is walking readers through how Transformer models actually work, step by step.

The site, Transformer Explainer, takes a 124-million-parameter model and breaks it down into its core parts: embedding, attention, and the MLP layer. The walk-through is built around the GPT-2 (small) model, which the explainer uses as a starting point for understanding the basics of text-generative Transformers.

Here is the bare structure of a Transformer, as the explainer lays it out:

Core Components

The explainer identifies three key parts of every text-generative Transformer:

1. Embedding, where text prompts are turned into numerical vectors that the model can understand.
2. Transformer Block, the fundamental building block that processes the input data.
3. Output Probabilities, where the model transforms processed embeddings into probabilities for the next token.

Tokenization

The explainer illustrates how prompts like “Data visualization empowers users to” are broken down into smaller pieces called tokens. Tokens can be whole words or parts of words. The explainer notes that GPT-2’s vocabulary has 50,257 unique tokens.

Attention and the MLP Layer

The Transformer block is where the model does its work. It includes the attention mechanism and the MLP layer. The attention mechanism allows tokens to communicate with each other, capturing contextual information and relationships between words. The MLP layer refines each token’s representation independently.

The explainer walks through multi-head self-attention step by step, showing how each token’s embedding vector is transformed into Query (Q), Key (K), and Value (V) vectors. It also explains that the self-attention mechanism is derived by multiplying the input embedding matrix with learned weight matrices.

The Numbers Behind the Model

The explainer’s decision to lean on GPT-2 (small) was a deliberate one. It is not the latest or most powerful Transformer model, but it shares the same architectural components and principles found in the current state-of-the-art models. The 124 million parameters are smaller than the models that actually run internet text generation today, which makes it an accessible entry point.

The explainer also notes that the embedding vectors are stored in a matrix of shape (50,257, 768), containing approximately 39 million parameters. That matrix allows the model to assign semantic meaning to each token, with similar tokens placed close together in high-dimensional space and dissimilar ones farther apart.

Long-Range Dependency

The explainer does make one clear point: the self-attention mechanism is the core innovation of Transformers. It allows the model to process entire sequences and capture long-range dependencies more effectively than previous architectures.

That is worth repeating. The attention mechanism is not a minor tweak. It is the reason Transformers have become the go-to architecture for deep learning models, powering text-generative models like OpenAI’s GPT, Meta’s Llama, and Google’s Gemini.

The explainer also explains the training process. It describes how the model learns its own positional encoding matrix from scratch, integrating it directly into the training process.

Key Facts

• Model: GPT-2 (small), 124 million parameters
• Vocabulary: 50,257 unique tokens
• Embedding matrix: (50,257, 768), approximately 39 million parameters
• Transformer blocks: 12
• Tokenization example: “Data visualization empowers users to”

The explainer is doing a job: making the inner working of Transformers accessible. It is a solid, well-structured piece of technical communication.

It is also, by design, incomplete. The explainer drops the question of training data entirely, and that is the point. The explainer is focused on how the model processes information, not where that information comes from. That is a reasonable choice for a walkthrough, but it is also a gap that the explainer itself leaves unfilled.

See the video the story is built around at poloclub.github.io.

AI · deep learning · GPT-2 · NLP · Transformers