Why hidden layers exist

The simplest model there is takes its inputs, multiplies each by a knob, adds them up, and squashes the total into a yes/no probability. On a map, whatever knob settings you pick, the boundary it draws between yes and no is always a straight line.

Straight lines handle plenty. They fail on a famous puzzle: the checkerboard, also called XOR (exclusive or). Bamboo grows in the top-right and bottom-left quarters, rocks in the other two. No single straight line can separate them.

Hit the wall

This network has no hidden layer: just the inputs x and y, straight to the answer. Train it and watch the accuracy.

It gets stuck. The accuracy settles around 75% at best, never solved. Why 75% and not 50%? A line can’t split four quarters correctly, but it can tilt so that three of the four land on the right side. That’s the ceiling, and no amount of training gets past it.

There are two ways out.

Way one: give it a better input

Tick the xy box, the two inputs multiplied together. In the top-right quarter x and y are both positive, so xy is positive. In the bottom-left both are negative, so xy is positive again. In the other two quarters xy is negative. Measured by xy, the checkerboard is a simple yes/no split, and a straight line handles it easily. That’s what the challenge above is asking for.

This was how most machine learning worked for decades: humans invented clever inputs (“features”) by hand.

Way two: let it invent its own

Untick xy and type 4 into Hidden layers. Now the inputs feed four neurons. Each one is its own little straight-line model, passing its answer through a bend (the activation, tanh or ReLU). The output combines those four bent answers. Bent lines combined can draw corners, curves and closed shapes. Train it.

The hidden layer has worked out something that does the job of xy without being told. That is the reason hidden layers exist: the network learns its own features. Stack more layers and later ones build features from the earlier ones’ features, which is what deep in “deep learning” means.

Now try the hardest shape. The spiral needs several layers or very well-chosen inputs: