Learning is turning knobs

“Machine learning” sounds mysterious. The idea underneath isn’t.

A model is a function: numbers go in, an answer comes out. Inside are knobs, numbers called weights that decide what the function does. A model with the wrong knob settings gives wrong answers. Training is the process of finding better settings, automatically, from examples.

It takes three ingredients:

1. Examples with answers. Here, dots on a map, each labelled bamboo (green) or rocks (grey).
2. A loss: one number that says how wrong the model is across all the examples. Zero would be perfect.
3. A way to improve. For each knob, work out which direction of turning would make the loss smaller, then turn every knob a small step that way. Repeat thousands of times.

That third step is gradient descent. The gradient is the list of “which way, and how steeply” for every knob at once. Calculus computes it exactly, so nobody has to guess. Picture walking downhill in fog: you can’t see the valley, but you can feel which way the ground slopes under your feet.

Watch it learn

The background colour is the model’s current opinion of every spot on the map: green means bamboo, grey means rocks, and paler means less sure. Press Train and watch the loss fall and the boundary bend into shape. Dots with a ring around them, drawn slightly smaller, are held back: the model never trains on them, and they’re what the accuracy number is measured on.

The learning rate

How big is each step? That’s the learning rate, and it’s one of the knobs you set, not the model.

• Too small: the loss creeps down so slowly you’d think nothing is happening.
• Bigger: faster, up to a point.
• Too large: each step overshoots the bottom of the valley, so the loss jumps around and can end up worse than where it started.

Try all three above, pressing New random knobs before each one. At 0.03 it creeps: the loss barely moves for the first few seconds, while the default is already done. At 3, on this small problem, it learns faster than the default. At 10, watch the loss leap and stumble. Where “too large” begins depends on the problem. On big models it starts much lower, which is why picking the learning rate is a real part of the job.