Seyed Masoud Hosseini · Overview · Study log · Ideas · Transcript · RSS feed

Neural Networks: Zero to Hero

Andrej Karpathy · AI · Fall 2026 · Planned

Lectures

  1. Lecture 1: The Spelled-Out Intro to Neural Networks and Backpropagation (2:25:52)
  2. Lecture 2: Building a Bigram Language Model (makemore, Part 1) (1:57:45)
  3. Lecture 3: Building an MLP Character-Level Language Model (1:15:39)
  4. Lecture 4: Activations, Gradients, and Batch Normalization (1:55:57)
  5. Lecture 5: Becoming a Backprop Ninja (1:55:24)
  6. Lecture 6: Building a WaveNet (56:21)
  7. Lecture 7: Let's Build GPT From Scratch (1:56:20)
  8. Lecture 8: State of GPT (42:40)
  9. Lecture 9: Building the GPT Tokenizer (2:13:34)
  10. Lecture 10: Reproducing GPT-2 (124M) from Scratch (4:01:26)

Notes

No notes yet.

References

No references yet.

Study log

Day one — building micrograd from scratch

Sep 15, 2026 · Neural Networks: Zero to Hero · 2.5 h

What I did

Watched the first lecture of Neural Networks: Zero to Hero and rebuilt micrograd alongside it: a Value class that records the operations applied to it, a topological sort, and a backward() pass that fills in gradients.

What I learned

  • Backpropagation is just the chain rule applied recursively over a DAG, in reverse topological order.
  • Gradients must accumulate (+=), not overwrite. A node used twice gets a contribution from each path.
  • A neuron is tanh(w · x + b); an MLP is layers of those. Everything else is bookkeeping.

Open questions

  • How do real frameworks avoid building a Python object per scalar?
  • Why does tanh fall out of favour against ReLU in deeper networks?