How a model learns.

Not a rule for every possible sentence. A set of learned settings—and a way to make the next guess better.

Before the hill, meet the knobs.

Weights are learned number settings that control how a network combines signals. Together with other learned settings, they are its parameters. Training adjusts them so predictions better match examples—often, for an LLM, predicting the next token (a word or word piece).

Why so many? Layers combine many signals with many others; the adjustable connections multiply. That gives the model room to learn many interacting patterns. It is not one weight per fact or concept, and more parameters alone do not guarantee a better model.

“A picnic on the river bank.”

“A deposit at the bank.”

Same learned weights
Different internal representationsContext changes the working values, not the shared settings.
Conceptual illustration, not actual measured embeddings.

Those input-dependent representations occupy what people often call latent or representation space: internal coordinates that can make useful relationships easier to work with. Think “a map of relationships,” not a filing cabinet. The weights help compute the map; the values change with the input. Context matters.

The hill below is a different space. It scores alternative weight settings, not the meanings of words. We are about to change the settings, not move a sentence around its representation.

Now give learning just two knobs.

Two-weight teaching model—not an LLM landscape

Each spot is a pair of weight settings. Height is their error, called loss. Take one step, then try a larger step. The goal is lower error—not a ball rolling downhill.

Possible weight settings
A map of loss for two weight settings Each line joins equal loss. A hollow circle is the start, a filled dot is now, and the arrow points opposite the gradient. -3 -30 03 3 weight x weight y Equal losson each line Start Now Downhilldirection
Ring: start. Dot: now. Arrow: downhill direction, normalized for visibility. Drag the 3D view to rotate, or focus it and use arrow keys. This is weight space, not latent space.

Static map. JavaScript adds a rotatable 3D view of the same model.

Two settings. One error score.

Take one step. The slope will tell us how to adjust both settings; the new height tells us whether that helped.

Starting error: 3.4.

Weight x
-2.2
Weight y
1.4
Updates
0
Did the error fall? At step zero the loss is 3.4. The horizontal axis counts updates; the vertical axis is loss in arbitrary units. 3.67 1.84 0 0 10 20 Error (loss)Updates
The same steps as the 3D view. Linear axes rescale to keep the actual errors visible.

Ready. Step makes one exact gradient-descent update; Play repeats it.

In the starter bowl, the ideal settings are zero. Loss is half the sum of the two squared setting errors. No text model is being trained. Changing a setting starts a fresh run; Reset repeats the selected run.

Go off-road: other surfaces, exact rates & every step

The first experiment changed only the step size. Here you can change the problem itself.

Choose the narrow valley to see a steep direction limit the safe rate; try the exact saddle to see why “stopped” is not “solved.”

One minimum. A modest rate shrinks both parameters; a rate above 2 makes nonzero starts grow.

L = (x² + y²) / 2
Analytic gradient: (x, y)

Current loss: 3.4. Gradient norm: 2.6077.

Analytic gradient: (-2.2, 1.4); central finite difference: (-2.2, 1.4). Numerical agreement is a check, not a proof of the derivative.

The map interpolates sampled contours; the 3D mesh uses a 76 × 76 grid over x and y from −3 to 3. Its vertical scale changes by surface. Only marked steps are optimizer states; connecting lines are not extra updates. Finite differences use displacement 0.00001.

Stops: gradient norm below 0.000001, 180 updates, a parameter exceeding 32, loss exceeding 10,000, or a nonfinite value. Divergence stops are explicit, not hidden clipping. This is exact full-gradient descent: no gravity, momentum, or random jitter.

Every evaluated step in this run
StepxyLossGradient xGradient y
0-2.21.43.4-2.21.4

What just happened?

The learner gets a direction, not the answer key.

The gradient describes how the error would change if each weight moved a little. Gradient descent moves opposite that local uphill direction. The learning rate sets how far.

That is why the too-large step is revealing. The direction is still locally downhill. The jump simply goes too far. “Try harder” is not an optimizer.

  1. 01 · ExamplesMake a predictionUse the current weights
  2. 02 · ErrorScore the missA chosen training objective
  3. 03 · GradientFind a directionBackpropagation computes it
  4. 04 · UpdateAdjust the weightsThe optimizer takes the step
Backpropagation computes gradients through the network using the chain rule. The optimizer decides how to use them. Different jobs; one training loop.

Learning is repeated adjustment. Understanding whether it worked is a separate job.

A lower error score is not a truth detector.

For a language model, predicting training text well is useful—but not the same as being reliable on your question. A system can fit what it saw and still fail on something new. The objective rewards what we chose to measure, not everything we hoped the model would become.

Real training has vastly more interacting parameters, estimates gradients from batches, and often uses more elaborate optimizers. This little hill does not promise a single best destination or a safe route to it. It makes one mechanism visible: error can guide a better next guess.

That repeated computation needs real machines. Next: what the useful answer must earn.

The derivatives, saddle point, and what AdamW changes

The bowl is L = (x2 + y2)/2, with gradient (x, y). Updates multiply each parameter by (1 − learning rate). From nonzero starts, rates strictly between 0 and 2 converge; 2 cycles; rates above 2 grow.

The valley is L = x2/2 + 6y2, with gradient (x, 12y). Convergence from all starts requires a rate strictly between 0 and 1/6. The direction with greater curvature sets that limit, even while the shallow direction moves slowly.

The two-well surface is L = (x2 − 1)2/4 + y2/2, with gradient (x3 − x, y). The origin has zero gradient but is a saddle, not a minimum. Minima at (−1, 0) and (1, 0) have zero loss. Exact zero in x stays zero; the simulation adds no secret jitter.

These are Euclidean gradients in chosen coordinates. Rescaling those coordinates changes the effect of a common learning rate. The height scale changes between surfaces only for display; the optimizer’s values are never silently clipped.

Real minibatch gradients are estimates and can vary between batches. That noise is not automatically helpful or a guarantee of escaping saddles or improving generalization. Adam adapts updates using moving estimates of gradient moments; AdamW decouples weight decay from the gradient-based update. Its paper reports particular empirical results, not universal guarantees. Neither optimizer is simulated here.

Sources, dates & methods

Original evidence review: . Second edition: . Individual source and review dates are listed below; none is a scenario forecast date.

All trajectories use explicit local analytic objectives, not a fitted language model. Central finite differences independently check their gradients numerically. The shared-weight/representation diagram is conceptual; it does not invent an embedding, assign a fact to a single weight, or claim that the loss surface is latent space.

  1. Neural networks: Nodes and hidden layers

    Google for Developers · Living documentation; reviewed 2026-11-02

    Shows how weights and biases combine inputs, how layers introduce more parameters, and why nonlinear operations matter. The two-weight lab is not a miniature transformer.

  2. Embeddings: Obtaining embeddings

    Google for Developers · Living documentation; reviewed 2026-11-02

    Distinguishes learned embedding parameters from representations and describes context-dependent embeddings. Real dimensions need not have interpretable names; the bank illustration is conceptual, not a measured embedding.

  3. Deep Learning, chapter 8: Optimization for Training Deep Models

    Ian Goodfellow, Yoshua Bengio & Aaron Courville · 2016

    Section 8.1 separates training objectives from generalization risk; minibatches and gradient-based optimization are discussed separately. This lab uses exact gradients of explicit teaching objectives.

  4. Deep Learning, chapter 6: Deep Feedforward Networks

    Ian Goodfellow, Yoshua Bengio & Aaron Courville · 2016

    Section 6.5 explains the chain rule and back-propagation. Computing gradients is distinct from choosing parameter updates.

  5. Decoupled Weight Decay Regularization

    Ilya Loshchilov & Frank Hutter · ICLR 2019; arXiv version 3, January 4, 2019

    Defines decoupled weight decay. Its reported image-classification experiments do not establish universal generalization or convergence guarantees for LLMs.

An independently built learning series. Models are teaching tools, not research findings or investment advice. No prompts or lab inputs are sent to an AI service.