A machine learning model is a formula with knobs. Learning means turning the knobs until the formula's guesses match a pile of examples. The computer scores how wrong it is with a single number, the loss, then uses calculus to work out which way to turn every knob to make that number smaller. Then it repeats. These three labs run that loop in your browser, from a straight line with two knobs to a small neural network with hundreds.
Say you want a straight line through some dots. A line has two knobs: its slope w and its height b where it crosses the y axis. Score any line by how badly it misses: square each miss, so misses above and below both count and big misses count extra, then average. That average is the loss. Now picture every possible pair of knob settings as a spot on a map, with the loss as the height of the ground. The best line sits at the bottom of a valley. Gradient descent finds it the way you would in fog: feel which way the ground slopes, step downhill, repeat. Drag the dots or tap empty space to add one. Tap the map to choose where the walk starts.
How to read this: Left: your dots and the current guess line. Right: every possible line is one spot on this map, and darker ground means a better fit. The amber trail is the walk downhill toward the gold ×.
Try this: Press Too big, then Run. The trail zig-zags and turns red. Then try About right.
The gradient is the pair of slopes: how fast the loss changes if you nudge w, and if you nudge b. It points uphill, so each step moves the knobs by minus the gradient, scaled by the learning rate η.
This loss is an exact bowl: its curvature H (the matrix of second derivatives) is the same everywhere. Along each of the bowl's two main axes, one step multiplies your distance from the bottom by 1 − η·λ, where λ is the curvature along that axis. Above η = 2/λmax that factor is bigger than 1 in size, so every step lands farther away than the last. Between 1/λmax and 2/λmax you overshoot and zig-zag but still settle.
A straight line has a formula for the exact answer (the gold line, via the normal equations), so here descent is only a demonstration. A neural network has no such formula. Descent is the only way down.
Uses the dots, start point, learning rate and step count on screen. The last comment shows what the page computed; the script prints the same numbers.
An artificial neuron is a few lines of arithmetic. It multiplies each input by a weight, adds the results plus a bias, and passes the total through a squashing curve called an activation. The output is a number between 0 and 1: how sure it is that a point is green rather than blue. One neuron can only split the plane with a straight line. Set its three knobs yourself and you can sort the two blobs, but no line sorts the XOR pattern. Wire neurons into layers and the boundary can bend. Nobody sets those hundreds of weights by hand. Backpropagation, the chain rule from calculus applied one layer at a time, computes the downhill direction for all of them at once.
How to read this: Each dot is an example, and its color is the right answer. The shading is the model's guess for every spot. Stronger shading means it is more sure. The gold line is where it is 50/50. On the right, circles are neurons and lines are weights. The bottom chart shows the loss after each pass over the data (an epoch). Lower is better, and each gridline is 10× smaller than the one above.
Try this: Pick XOR and try to sort it with the three sliders. You can't. Then switch to Network · it learns and press Train.
The loss, called cross-entropy, punishes confident wrong answers hardest: predicting 0.01 for a green point costs −log 0.01 ≈ 4.6. Its derivative with respect to the output neuron's raw total comes out as simply p − y, the size of the mistake.
That error is passed backward. Each weight gets blamed in proportion to the input it saw, and each hidden neuron gets the blame from the layer above, weighted by its connections and scaled by its activation's slope f′. For tanh f′ = 1 − a², for sigmoid a(1 − a), for ReLU 1 or 0. One backward pass costs about as much as two forward passes, whether the network has ten weights or a billion.
One epoch here is one pass over all 200 points followed by one step. The step uses momentum: the velocity v remembers 90% of the previous step, which carries the walk through flat stretches.
Same layers, activation, learning rate, momentum and initialization scale as the page. numpy draws different random numbers from the page's seeded generator, so its epoch-by-epoch losses differ slightly.
A model that aces its practice examples can still be useless. Here the hidden rule is a smooth curve (gold), and you only get a dozen noisy measurements of it (green dots). Fit a polynomial to them. Degree 1 is a straight line, degree 3 can bend twice, and degree 11 has twelve knobs, enough to pass through all twelve dots exactly. The fair test is fresh data the model never saw (the faint dots). Raise the degree: practice error falls to zero while test error climbs. That is overfitting, memorizing the noise instead of learning the rule. Ridge regularization charges a fee for large coefficients, which keeps the curve calm.
Try this: Drag Degree to 11 and watch the amber curve hit every green dot while turning red between them. Then raise Ridge λ.
Every measurement here carries random noise with standard deviation 0.2, so even the true curve scores a test error near 0.2² = 0.04 (the dashed line in the lower chart). Nothing can beat that floor. A training error far below it means the model has fit the noise itself, and noise does not repeat in new data.
The cure is a trade. A stiff model misses real bends (underfitting); a floppy one chases noise (overfitting). Ridge keeps the floppy model but makes wild coefficients expensive. The intercept c₀ is left free so the fee never drags the whole curve toward zero. More data helps too: slide the training dots up and the high-degree fits calm down.
The solve is exact, with one numerical precaution. x is kept in [−1, 1] so every power stays at most 1. On [0, 100], x¹¹ would reach 10²², the columns of V would be nearly parallel, and the computer would lose most of its 16 significant digits. The page solves the stacked system by Householder QR rather than forming VᵀV, which would square that sensitivity.
The same three ideas scale up to the systems in C06 · Embeddings & vector search and C07 · How a language model writes: billions of knobs, trained by the gradient descent of Lab 1 with the backpropagation of Lab 2, checked against text held out from training.
The training dots, degree and λ are the ones on screen, so the coefficients and training error match the page exactly. The test dots come from numpy's generator here, so that number differs slightly.