Transformer
A transformer is the neural network architecture that underlies modern large language models, introduced in the 2017 paper "Attention Is All You Need" by Vaswani et al.
Definition¶
A transformer is a type of neural network architecture designed to process sequential data — most importantly, text. Transformers replaced earlier recurrent architectures (RNNs, LSTMs) as the foundation of natural language processing by introducing the attention mechanism, which allows the model to consider relationships between all parts of an input simultaneously rather than processing it word by word.
The Attention Mechanism¶
The core innovation of transformers is self-attention: for each token (word or subword) in the input, the model calculates how much "attention" to pay to every other token when generating a representation of the current one. This allows the model to understand that in "The bank by the river was steep," "bank" refers to a riverbank — because "river" draws high attention — rather than a financial institution.
Attention operates in parallel across all token positions, which makes transformers faster to train than sequential architectures and better at capturing long-range dependencies in text.
Transformers and Large Language Models¶
GPT (Generative Pre-trained Transformer), BERT, Claude, Gemini, and Llama are all transformer-based models. The "T" in GPT stands for transformer. The architecture has become so dominant that "transformer model" and "large language model" are now almost synonymous for text-based AI.
Practical Implications¶
The quality improvements in AI writing since 2018 are largely attributable to scaling transformer architectures: more parameters, more training data, more compute. Understanding that these models work through statistical pattern matching — not reasoning or understanding — helps set appropriate expectations for their outputs and limitations.