How Do Language Models Think? A Guide to Prediction Styles
- Causal (left-to-right) prediction — what TinyLlama/GPT/Llama do The model reads tokens in order and, at each position, can only look at tokens before it — never after. It predicts the next token, one at a time.
"The cat sat on the ___"
↑ can see all of this → predicts "mat"
` This is enforced with a causal mask — a technical trick that blocks the model from "cheating" by peeking at future tokens during training. This matters because at actual use-time (inference), the future tokens genuinely don't exist yet — you're generating them one by one. So training has to match how it will actually be used. Why it's the standard choice for generation: if you want a model that can write open-ended text, chat, or continue a story, it must work this way — you can't generate word 5 by looking at word 7, because word 7 doesn't exist yet.
- Masked (bidirectional) prediction — what BERT does Instead of predicting the next word, you take a complete sentence, hide (mask) some random words in the middle, and ask the model to fill them in — using context from both sides.
"The cat [MASK] on the mat"
↑ before ↑ after
both used to predict "sat"
This produces excellent understanding of text (great for classification, search, embeddings — actually close to what your FastEmbed models like all-MiniLM-L6-v2 and bge-m3 are doing internally). But it's not naturally built for generating long text, because it was never trained to produce text one token after another in order — it was trained to fill in blanks in text that already exists. Rule of thumb: bidirectional/masked models → good at understanding/embedding text. Causal/left-to-right models → good at generating text.
- Encoder-decoder (sequence-to-sequence) — the original 2017 transformer, and models like T5 This is actually two models glued together:
- Encoder reads the entire input (bidirectionally, like BERT) and builds a rich understanding of it.
- Decoder then generates the output left-to-right (causally, like GPT) — but at each step, it's also allowed to "look back" at the encoder's understanding of the full input.
This was designed for translation: e.g., read a full French sentence (encoder, sees everything), then generate the English sentence word-by-word (decoder), while constantly referring back to the French meaning.
Encoder (bidirectional): "Le chat est assis" → full understanding
Decoder (causal): "The" → "cat" → "is" → "sitting"
(each step looks at encoder output too)
GPT/Llama/TinyLlama threw away the encoder half because for pure text generation (no separate "source" language/document to translate from), you don't need a whole second network just to encode an input — the decoder alone reading everything left-to-right does the job.
Why causal won for today's LLMs
Even though masked/bidirectional models "see more" per prediction, causal models turned out to scale better with size and data, and they map directly onto the actual generation task (predict the next thing, repeatedly) — which is what chat, coding assistants, and completion tools need. That's why essentially every major LLM you interact with day-to-day (GPT, Llama, TinyLlama, Claude included) is a decoder-only causal model, while masked models like BERT live on mostly as embedding/classification backbones — which is actually relevant to your project, since your embedding models for duplicate detection are exactly this "understanding-focused, bidirectional" family, not the causal-generation family.