About TinyLlama & architecture resources

I have started to learn about this model (TinyLlama), and I have come across interesting facts and data that I want to share with you, dear readers.

For training this model (TinyLlama), we have two huge datasets: general information and code information, which are SlimPajama and Starcoderdata. The first one is about general data on the internet from websites like Wikipedia and other useful sources, and the second one is about code and knowledge of basic programming. It brings this model forward in two ways (general knowledge and code knowledge).Based on the documentation, the model was trained on 950B tokens, and we had 3 epochs for training. Each epoch over 950B tokens costs significant GPU time (TinyLlama already took about 90 days).

22 transformer blocks and 16 attention heads. When we have input data, it passes through 22 blocks, meaning the data moves through all these layers step by step. In each block, we have 16 attention perspectives, which means the data in each block is processed in 16 different dimensions.Inside each of those 22 blocks, attention isn’t done just once — it is split into 16 parallel parts, called heads. Each head independently asks: "for this word, which other words in the sentence matter most?" But each head can specialize in a different kind of relationship. For example (illustrative, not exact):

Head 3 might mostly track "which pronoun refers to which noun

• Head 9 might mostly track nearby-word grammar
• Head 14 might track long-range topic relevance

All 16 heads run at the same time on the same input, then their outputs get combined (concatenated + mixed) into one result before moving to the next block.

One transformer block:

Input → [Head1, Head2, ..., Head16] (all run in parallel) → combine → feed-forward → output

Putting them together

• 22 = how many times this attention-plus-processing step repeats, stacked (depth)
• 16 = how many parallel perspectives attention takes at each single step (width)

BPE tokenizer Another thing we have to talk about is BPE, and now we are going to explain it. It is an algorithm that builds a vocabulary by starting with single characters and repeatedly merging the most frequent pairs into new tokens until it reaches a target size (e.g., 32,000).It means the model starts to find the most frequent pairs, like ‘-est’ at the end of ‘strongest’, ‘lowest’, and ‘widest’. It helps the model understand each token within words more efficiently. At the end, we have a vocabulary that includes the most frequent tokens, and then we use it as a guide for new words. Result: common words become single tokens ("the", "and"), while rare/made-up words get split into smaller familiar pieces. This is why the model can still handle words it's never seen before — it breaks them into known fragments.

RoPE (Rotary Position Embeddings) Transformers see all tokens at once, so they need a way to know word order (attention alone doesn't know "dog bites man" from "man bites dog"). RoPE encodes each token's position by mathematically "rotating" its number vector by an angle based on its position in the sentence. Nearby positions get similar rotations, far-apart positions get very different rotations — so the model can infer relative distance between words cheaply.

SwiGLU activation An "activation function" is the small nonlinear step inside each layer that lets the network learn complex, non-straight-line patterns (without it, stacking layers would be pointless — it'd collapse to one big linear equation).

more data : SwiGLU was introduced by Noam Shazeer (Google) in a 2020 paper, "GLU Variants Improve Transformer.

Later, we will explain all the above topics in more detail, as they can be difficult to understand at first glance.