Before the Boom

Technical History · Part I

A theoretical and technical prehistory of modern language models, from symbolic reasoning, backpropagation, and representation learning to recurrent networks, sequence modeling, and attention.

Modern language models are often described as the product of the Transformer architecture combined with computation at scale. This account is not wrong, but it can obscure a more important fact: the architectural breakthrough of 2017 did not appear in isolation. It depended on a collection of theoretical objects and engineering components developed over several decades, including differentiable computation, backpropagation, distributed representations, probabilistic language modeling, stable optimization, recurrent states, gated memory, encoder–decoder architectures, and content-addressed attention.

This article traces the technical lineage that preceded the Transformer. Its purpose is not merely to list milestones. Instead, it asks three questions of each stage: What difficulty was it designed to solve? What mathematical structure did it introduce? Why did that structure become an essential part of later language models?

The history of the Transformer should not begin with self-attention. It should begin with the problem of expressing intelligence as an optimizable computation graph.

1. Two Traditions of Artificial Intelligence: Rules and Learning

1940s–1980s · Symbolic systems and early learning

Early artificial intelligence developed primarily along two lines. Symbolic AI represented knowledge through symbols, propositions, and rules, and then derived conclusions through search and logical inference. Connectionism instead sought to make systems adjust the strengths of their connections from data, allowing internal representations to emerge from experience.

In a symbolic system, a typical problem can be written as state-space search. Let \(\mathcal{S}\) denote the set of states, \(\mathcal{A}(s)\) the actions available in state \(s\), and \(T(s,a)\) the transition function. The objective is to find a path from an initial state \(s_0\) to a goal set \(\mathcal{G}\). With an additional utility function, rational decision-making can be expressed as

Expert systems encoded knowledge as large collections of if–then rules. Given a knowledge base \(K\) and a query \(q\), an inference engine attempted to determine whether \(K\models q\). This approach succeeded in domains with clear boundaries and stable rules, but it also revealed three persistent problems: knowledge acquisition was expensive, manually written rules could not cover open environments, and the resulting systems were brittle under distribution shifts and exceptions. Several downturns in AI during the 1970s and 1980s arose partly from the gap between promised capabilities and actual scalability.

Learning-based methods adopted a different position. Rather than requiring humans to specify every rule, they searched for parameters \(\theta\) that minimized empirical risk on a dataset \({(x_i,y_i)}_{i=1}^{N}\):

2. From Artificial Neurons to the Perceptron

McCulloch and Pitts (1943) abstracted a neuron as a weighted computational unit with a threshold. Given inputs \(x_1,\ldots,x_d\), its output is

The importance of this model does not lie in its biological realism. It supplied a composable computational primitive: a linear transformation followed by a nonlinear function. Fully connected layers, convolutional layers, and the projection matrices used in attention all inherit this basic structure.

Rosenblatt’s perceptron (1958) added a data-driven rule for updating the weights. For binary classification, one common form is

When the data are linearly separable, the perceptron update converges to a separating hyperplane. A single-layer linear classifier, however, cannot represent nonlinear relations such as XOR. This limitation showed that genuinely complex representations require multiple composed layers. That requirement immediately raised another question: how should each layer know how to modify its own parameters?

3. Multilayer Networks and Backpropagation

A multilayer perceptron composes simple transformations layer by layer. Let \(h^{(0)}=x\). The \(\ell\)-th layer is

Depth allows a model to construct hierarchical representations, but it also creates complex dependencies among parameters. Rumelhart, Hinton, and Williams (1986) popularized the use of backpropagation to learn internal representations. Its mathematical basis is the chain rule. If the loss is \(\mathcal{L}\), adjacent layers satisfy

Backpropagation is not a particular network architecture. It is a mechanism for credit assignment: it determines which intermediate computations are responsible for a prediction error and how much each parameter should change. Modern automatic-differentiation systems perform the same operation, while organizing local derivatives over complex computation graphs into efficient programs.

4. Why Deep Networks Are Difficult to Train

If a network or recurrent system spans \(T\) computational steps, its gradient typically contains a product of Jacobian matrices:

When the dominant singular values of these matrices remain below one, the gradient approaches zero; when they remain above one, it grows rapidly. These are the vanishing- and exploding-gradient problems. The former prevents early layers and distant states from learning effectively, while the latter makes optimization numerically unstable.

Gradient clipping as a constrained update

After backpropagation, collect all parameter gradients into one vector

Global norm clipping constrains the update to an \(L_2\) ball of radius \(\tau\). It can be written as the Euclidean projection

If \(g\) already lies inside the ball, the projection leaves it unchanged. Otherwise, the nearest feasible point lies on the same ray as \(g\), but has norm \(\tau\):

In PyTorch, clipping is applied after gradients have been computed and before the optimizer updates the parameters:

optimizer.zero_grad()
loss.backward()

total_norm = torch.nn.utils.clip_grad_norm_(
    model.parameters(),
    max_norm=1.0,
)

optimizer.step()

Here, total_norm is the total gradient norm before clipping, so it can also be logged as a diagnostic signal.

Measuring gradient gain across layers

The same Jacobian view gives a more local description of vanishing gradients. Let

Backpropagation through one layer gives

Define the gradient gain at layer \(k\) as

Across \(K\) layers, these local gains multiply:

Persistent values \(\gamma_k<1\) produce exponential attenuation, while persistent values \(\gamma_k>1\) produce exponential amplification. This also explains why an isolated small gradient is not sufficient evidence of vanishing gradients: the relevant signal is the systematic reduction of gradient norm across depth or time.

As an illustrative regularizer, one could penalize deviations of the local gain from one:

This objective encourages norm preservation, but it should be interpreted as a diagnostic construction rather than a standard remedy. Computing it exactly may require higher-order differentiation, and forcing every layer to preserve gradient norm can interfere with useful task-specific dynamics. Initialization, gating, normalization, and residual paths usually address the same problem more directly.

A collection of training components developed after 2010 gradually transformed models that were merely expressive in theory into systems that could be optimized in practice.

Two routes from SGD to Adam

Adam is easiest to understand not as an isolated algorithm, but as the point where two lines of optimizer research meet. Both begin with stochastic gradient descent:

The two lines address different weaknesses of this update:

1. Adaptive-step route: How should each parameter receive a learning rate appropriate to its own gradient scale? A useful conceptual lineage is Rprop → AdaGrad → RMSProp.
2. Momentum route: How should updates accumulate consistent directions while suppressing noisy oscillation? This leads from classical momentum to Nesterov-style look-ahead and exponential first-moment estimation.

These are conceptual rather than strict genealogical arrows. AdaGrad is not algebraically derived from Rprop, for example, but both replace one global step rule with parameter-wise adaptation.

Route I: adaptive learning rates

Rprop: adjust each step from sign consistency. A single learning rate \(\eta\) treats all coordinates equally even when their gradient scales differ substantially. Rprop takes an early coordinate-wise approach. It discards gradient magnitude and uses only the sign of each coordinate:

The step size grows when consecutive gradients agree in sign and shrinks when they disagree:

where \(\eta_{+}>1\) and \(0<\eta_{-}<1\). Repeated agreement suggests that movement along that coordinate is stable; a sign reversal suggests that the previous step may have crossed a local optimum.

AdaGrad: normalize by cumulative squared gradients. AdaGrad uses gradient magnitude to construct a separate effective learning rate for each coordinate. With all products and square roots interpreted elementwise,

Coordinates with consistently large gradients acquire a larger denominator and therefore a smaller effective step. However, because \(G_t\) only increases, AdaGrad’s effective learning rates can eventually become extremely small.

RMSProp: replace the full history with a moving second moment. RMSProp replaces the unbounded cumulative sum with an exponential moving average:

The common principle behind these methods can be derived by asking that the root-mean-square update of coordinate \(i\) remain near a target scale \(\eta\):

If \(\Delta\theta_i=-\alpha_i g_i\), then

The algorithms on this route differ largely in how they estimate the second moment \(\mathbb{E}[g_i^2]\): AdaGrad uses a cumulative history, while RMSProp uses an exponential moving average that can forget obsolete gradient scales.

Route II: momentum and direction smoothing

The adaptive-step route controls how far to move along each coordinate. Momentum addresses a different question: which direction remains reliable across multiple noisy minibatches? In a form aligned with Adam’s notation,

Unrolling the recurrence shows that momentum is an exponentially weighted average of past gradients:

Gradients that repeatedly point in the same direction accumulate, while rapidly alternating components partially cancel. Momentum therefore acts as a low-pass filter: it accelerates movement along stable directions and reduces oscillation along high-curvature directions.

Classical momentum is often written as \(u_t=\mu u_{t-1}+g_t\), followed by \(\theta_t=\theta_{t-1}-\eta u_t\). This differs from the normalized exponential-average form only by a constant rescaling that can be absorbed into the learning rate.

Nesterov momentum refines the same route by evaluating the gradient near a look-ahead position:

The look-ahead gradient can correct the trajectory before the full momentum step is taken. Vanilla Adam does not directly contain the complete Nesterov update, but it inherits the broader idea of stabilizing direction through an exponential first-moment estimate.

Adam: the merger of the two routes

Conceptually, Adam combines

Its update can be written as

The correction factors arise because both moving averages are initialized at zero. Assume for the moment that gradients have a stationary mean \(\mathbb{E}[g_t]=\mu\) and that \(m_0=0\). Expanding the recurrence gives

Taking expectations and evaluating the geometric series,

Thus \(m_t\) systematically underestimates the first moment early in training. Dividing by \(1-\beta_1^t\) removes this initialization bias:

The same argument applied to \(v_t\), under the stationary approximation \(\mathbb{E}[g_t^2]=\nu\), yields

Bias correction matters most during the first updates, when \(\beta_1^t\) and \(\beta_2^t\) are not yet negligible. Adam therefore combines a direction-smoothed first moment with a scale-normalizing second moment while correcting the transient bias introduced by zero initialization.

Knowledge distillation: why temperature is squared

Not every important pre-Transformer development was aimed at making networks deeper. Knowledge distillation (Hinton, Vinyals, and Dean, 2015) addressed a different problem: how to transfer the behavior of a large teacher or ensemble into a smaller student. Its central claim is that knowledge is present not only in the correct label, but also in the teacher’s relative preferences among incorrect classes.

Let \(v_i\) denote the teacher logit for class \(i\), \(z_i\) the corresponding student logit, and \(T>0\) the temperature. Their softened distributions are

At \(T=1\), these are ordinary softmax probabilities. Increasing \(T\) produces a higher-entropy distribution, making small differences among non-target classes easier to observe. Those relative probabilities convey what is often called the teacher’s dark knowledge: for example, that an image classified as a BMW is more similar to another car than to a carrot, even when both alternatives receive low probability.

The student is commonly trained with both a soft-target loss and the ordinary hard-label loss:

A standard combined objective is

To see this, first differentiate the log-softmax:

Because the teacher probabilities sum to one,

This gives one explicit factor of \(1/T\). A second factor appears because the teacher and student distributions themselves become closer as the temperature rises.

Softmax is invariant to adding the same constant to every logit, so define centered logits

for which \(\sum_i\tilde z_i=\sum_i\tilde v_i=0\). When \(T\) is large relative to the logit magnitudes,

For \(N\) classes, the centered first-order terms cancel in the denominator, giving

Therefore,

and the soft-loss gradient becomes

A binary example makes the effect concrete. Let the teacher logits be \(v=[2,0]\) and the student logits be \(z=[1,0]\). At \(T=1\),

At \(T=10\), the distributions are much softer:

so

The exact finite-temperature ratio need not be precisely \(T^{-2}\), but the high-temperature derivation explains the dominant scaling. Multiplying by \(T^2=100\) restores the gradient to the same general range as the hard-label signal. At inference time, the distilled student returns to \(T=1\).

These components should not be treated as isolated tricks. Initialization determines signal scale at the start of training; activation functions determine local derivatives; normalization and residual connections alter conditioning across depth; and the optimizer determines how gradients accumulate into parameter updates. The later ability to stack many Transformer blocks depended on this entire optimization infrastructure.

5. Convolutional Networks and the Revival of Deep Learning

Convolutional networks were not direct predecessors of language models, but they demonstrated that end-to-end representation learning could scale reliably with large datasets and GPUs. A two-dimensional convolution has the general form

Local connectivity and weight sharing encode translation structure while greatly reducing the number of parameters. From LeNet to AlexNet (2012), computer vision established an engineering paradigm later inherited by language modeling: large datasets, GPU training, end-to-end objectives, deep representations, and reproducible optimization recipes.

The significance of AlexNet therefore extends beyond image classification. It showed that when model architecture, data scale, computing platforms, and training methods advance together, neural networks can replace traditional pipelines built around manually engineered features.

6. Language Models: From Counts to Continuous Representations

The purpose of a language model is to assign a probability to a sequence of symbols \(w_{1:T}\). By the chain rule of probability,

Traditional \(n\)-gram models use a finite-order Markov assumption:

Their parameters are usually estimated from corpus counts, with smoothing used to handle unseen combinations. Their fundamental difficulty is the sparsity of discrete contexts: two contexts that are semantically similar but lexically different cannot naturally share statistical strength.

Bengio et al. (2003) introduced a neural probabilistic language model that maps each word to a continuous vector and uses a feed-forward network to predict the next word. Given an embedding matrix \(E\), a fixed-window context can be written as

Maximum-likelihood training is equivalent to minimizing token-level cross-entropy:

The corresponding perplexity is

Continuous representations allow contexts with similar semantic or syntactic roles to share parameters. This was the decisive transition from count-based statistical language models to neural language models. A fixed context window, however, still cannot represent dependencies of arbitrary length.

7. Recurrent Networks: Compressing History into State

A recurrent neural network processes a variable-length sequence through a recursively updated state:

The state \(h_t\) is expected to summarize all preceding inputs. This is an important generalization beyond fixed-window models. Training, however, requires backpropagation through time, so its gradients contain long products of Jacobians. Information that a model can preserve in principle is not necessarily information that optimization can learn in practice.

LSTM: Explicit Control over Writing and Forgetting

Hochreiter and Schmidhuber (1997) introduced the LSTM, which controls information flow through gates. A common modern formulation is

The forget gate \(f_t\) determines how much old memory to preserve, the input gate \(i_t\) determines how much new content to write, and the output gate \(o_t\) determines how much internal memory to expose as the current state. The most important feature is the additive update of the cell state. Compared with the repeated nonlinear transformations of a standard RNN, it provides a more direct path for gradient propagation.

LSTMs remained central to speech recognition, machine translation, and language modeling for many years. Yet they retained an unavoidable sequential dependency: \(h_{t-1}\) must be computed before \(h_t\). This dependence later became a major obstacle to large-scale parallel training.

8. Word Vectors and Distributed Semantics

The distributional hypothesis holds that a word’s meaning can be characterized by the contexts in which it occurs. Neural language models already learned word vectors implicitly, while Word2Vec (2013) reduced representation learning to an efficient local prediction task.

The idealized skip-gram objective is

To avoid computing a softmax over the entire vocabulary, negative sampling turns each update into a binary classification problem:

Here \(w_c\) is the center word, \(w_o\) is an observed context word, and \(w_k\) is a negative sample. This method made large-scale word-vector training economical and established semantic relations in vector spaces as a standard representational framework in NLP.

Its limitation is equally clear: each word usually receives only one static vector. The meaning of the same word therefore cannot change across sentences. Contextual representations require a sequence model to participate in the computation.

9. Sequence-to-Sequence Learning: From Representing Sequences to Generating Them

Sequence to Sequence Learning (2014) used one recurrent network to encode an input and another to generate an output autoregressively. For an input \(x_{1:S}\) and output \(y_{1:T}\), the model is

The importance of this architecture is that it does not require separate output spaces for translation, summarization, or question answering. Instead, it unifies these tasks as conditional sequence generation. The entire input, however, is compressed into a single vector \(c\), creating a severe information bottleneck. As the input grows longer, it becomes increasingly difficult for the encoder to preserve every relevant detail in one fixed representation.

10. Attention: From Fixed Compression to Content Addressing

Bahdanau, Cho, and Bengio (2014) proposed reading all encoder states anew when generating each output symbol. Let \(s_{t-1}\) be the previous decoder state and \(h_j\) the \(j\)-th encoder state. The alignment score is

Attention solved two important problems. First, information no longer had to pass entirely through the encoder’s final state. Second, the model could learn a soft alignment between input and output. At this stage, however, attention was still attached to an RNN encoder and an RNN decoder. State updates remained sequential, and long-range information still had to travel partly through recurrent paths.

11. By 2016, the Necessary Components Were in Place

By 2016, most of the foundations of modern language models had already appeared:

1. Probabilistic objective: A language sequence could be modeled as a product of conditional probabilities and trained with cross-entropy.
2. Differentiable representations: Words, contexts, and hidden states could be mapped into continuous vector spaces.
3. Credit assignment: Backpropagation could train computation graphs composed of many differentiable modules.
4. Stable deep training: Initialization, non-saturating activations, normalization, residual connections, gradient clipping, and adaptive optimizers had matured.
5. Sequential state: RNNs and LSTMs could represent variable-length histories and gated memory.
6. A unified generation interface: Sequence-to-sequence learning converted multiple NLP tasks into conditional generation.
7. Content addressing: Attention allowed a model to access different input positions according to its current need.
8. Open-vocabulary modeling: BPE subword segmentation (2016) established a compromise between characters and words, reducing both out-of-vocabulary failures and the need for extremely large vocabularies.

12. Timeline of Key Milestones

Conclusion

The history before the Transformer shows that artificial intelligence did not advance through a simple succession in which one architecture replaced another. A more accurate account is that problem formulations, mathematical tools, and engineering mechanisms accumulated over time and were eventually recombined. Symbolic AI supplied the language of states, actions, planning, and structured reasoning. Statistical learning established the principle of estimating models from data. Neural networks provided differentiable representations. Backpropagation solved internal credit assignment. The infrastructure of deep learning made large-scale optimization practical. RNNs, LSTMs, sequence-to-sequence learning, and attention progressively defined the central problems of sequence modeling.

The value of understanding this history is not merely to identify who introduced a particular module first. It allows us to distinguish advances caused by new modeling assumptions from those enabled by improved optimization, greater computational and data scale, or the recombination of existing ideas in a new system. The same distinction remains essential when evaluating research on large language models today.