Autograd Lab

A reverse-mode automatic differentiation engine written from scratch — no PyTorch, no libraries — training a real neural network live in your browser. Every gradient you see below was computed by ~120 lines of hand-written chain rule.

reverse-mode autodiff tanh MLP · SGD 0 dependencies view source — it's all here

CONTROLS

—
loss (MSE)
—
accuracy
0
steps
—
‖grad‖

GRADIENT FLOW · first layer w→n0

DECISION BOUNDARY — live

class +1 class −1 background = model's output (blue: +, red: −, pale: undecided)

LOSS CURVE

THE NETWORK — edge weight = |w|, colour = sign

WHAT IS AN AUTOGRAD ENGINE?

Every scalar in the network is a Value that remembers how it was created. Arithmetic (+, *, tanh…) builds a computation graph. Calling loss.backward() walks the graph in reverse topological order and applies the chain rule at every node — that's all backpropagation is.

THE CHAIN RULE, BY HAND

For c = a · b, the engine closes over:

  1. ∂c/∂a = b → a.grad += b.data * c.grad
  2. ∂c/∂b = a → b.grad += a.data * c.grad

The whole engine is src/engine.js — about 120 lines, view source.

WHAT TO TRY

  • Set LR to 1.0 on spiral — watch the loss explode. Classic.
  • Drop hidden layers to 0 — the net can't bend, so moons stay unsolved.
  • Crank noise to 0.5 and add L2 — watch it resist memorising.
  • Pause mid-training and inspect the gradient table (left panel).