Lecture 15: Efficient Methods and Hardware for Deep Learning

What this lecture covers

Guest lecturer Song Han addresses a practical problem behind the models covered earlier in the course: state-of-the-art networks keep growing in size, training time, and energy cost, which makes them hard to deploy on phones, embedded devices, and data centers. The lecture organizes solutions into a two-by-two grid of algorithms and hardware for inference and for training, arguing that the best gains come from co-designing both together rather than tuning hardware for fixed algorithms.

After watching, you can explain how pruning, weight sharing, quantization, and structural tricks like Winograd convolutions shrink and speed up trained networks, how specialized hardware such as Google's TPU and Han's own EIE accelerator exploit those compressed representations, and how training-side techniques such as data/model parallelism, mixed-precision arithmetic, and model distillation cut training cost. It closes with a tour of recent GPU and TPU hardware built around these ideas.

Key ideas

• Pruning: removing low-magnitude weight connections and retraining the remainder can shrink parameter counts by roughly 10x with little to no accuracy loss.
• Weight sharing: clustering similar weights to a small set of centroids (via k-means) lets each weight be stored as a short index instead of a full-precision number, cutting bits per parameter.
• Deep Compression: combining pruning, weight sharing, and Huffman coding compresses networks 10x to 49x without hurting accuracy.
• Quantization: reducing numeric precision (for example FP32 to INT8) cuts energy and chip area cost roughly proportionally, since memory access dominates energy use over arithmetic.
• Winograd convolution: an equivalent but algebraically different way to compute convolutions that reduces the number of multiplications needed.
• EIE (Efficient Inference Engine): specialized hardware designed by Han that computes directly on pruned, weight-shared, sparse models, skipping zero-valued weights and activations.
• Mixed-precision training: using FP16 for multiplication and FP32 for accumulation and weight updates, giving roughly the memory and energy savings of FP16 while preserving FP32-level accuracy.
• Model distillation: training a small student network on the softened output probabilities of larger teacher networks so it approaches the teacher's accuracy on far less data.

Walkthrough

Why models are getting harder to deploy (1:12)

The lecture opens with the trend of model size and training cost growing rapidly (for example, ImageNet-winning model size increasing 16x from 2012 to 2015), and identifies three resulting challenges: model size limits mobile deployment and over-the-air updates, training speed limits research iteration, and energy consumption drives real costs, illustrated by the electricity cost of training AlphaGo. It traces the energy cost to memory access, which is far more expensive than arithmetic operations, motivating algorithm-hardware co-design instead of designing hardware around fixed algorithms.

Hardware building blocks (4:15)

A brief primer distinguishes general-purpose hardware (latency-oriented CPUs versus throughput-oriented GPUs) from specialized hardware (reprogrammable FPGAs versus fixed-logic ASICs like Google's TPU), and explains floating-point and integer number representations, showing that halving bit-width from 32 to 16 bits cuts both energy and chip area roughly 4x.

Pruning, weight sharing and Deep Compression (9:31)

Pruning removes redundant connections identified by small weight magnitude, and iterative pruning-and-retraining can remove up to about 90% of parameters without hurting accuracy, on both convolutional networks and RNNs/LSTMs. Weight sharing then clusters remaining weights so each can be represented by a short index into a small codebook. Combining pruning, weight sharing, and Huffman coding (Deep Compression) compresses networks 10x to 49x, and applying it to the already-compact SqueezeNet architecture yields models under half a megabyte with AlexNet-level accuracy, producing measured 3x to 5x speedups and 3x to 6x energy efficiency gains across CPU, GPU, and mobile GPU.

Quantization and structural compression (20:44)

The lecture covers linear quantization to low bit-widths (used in TPUs), low-rank approximation that factors convolution and fully connected layers into smaller operations, and extreme ternary/binary weight networks that keep only a scaling factor and a small set of discrete weight values while training with full precision. Winograd transformation is presented as an algebraically equivalent way to compute convolutions with fewer multiplications, already adopted in cuDNN.

Hardware for efficient inference: TPU and EIE (28:53)

Google's TPU is described as an ASIC built around a large INT8 matrix multiplication unit and a sizeable on-chip software-managed buffer, achieving high peak throughput but often falling well below peak in practice due to memory bandwidth limits, illustrated using a roofline model that separates compute-bound from memory-bound regimes. EIE, Han's own accelerator, is designed to compute directly on sparse, compressed, weight-shared models, skipping zero weights and activations, achieving large reported speedups and energy efficiency gains over CPU, GPU, and other ASICs.

Efficient training: parallelism and mixed precision (42:09)

Turning to training, the lecture covers data parallelism (multiple machines training on different data with coordinated weight updates through a parameter server), model parallelism (splitting a model or its input across processors), and hyperparameter parallelism. It then explains mixed-precision training in detail: using FP16 for the forward and backward computation but FP32 for weight updates, which NVIDIA showed converges to accuracy comparable with full FP32 training.

Model distillation and Dense-Sparse-Dense training (50:20)

Model distillation trains a smaller student network on the softened probability outputs of larger teacher networks rather than hard labels, letting the student approach teacher accuracy with much less data. Han's own Dense-Sparse-Dense training re-densifies a pruned network and retrains it as a regularization technique, giving reported accuracy gains of about 1% to 4% on ImageNet and improving qualitative results on image captioning.

Hardware for efficient training: Volta and Cloud TPU (54:27)

The lecture surveys recent training hardware: NVIDIA's Pascal and newly released Volta GPUs, the latter introducing Tensor Cores that perform 4x4 mixed-precision matrix multiply-accumulate operations in a single cycle, giving measured speedups over Pascal on ResNet-50 training and inference throughput. It also covers Google's Cloud TPU, which extends the original inference-only TPU to support training, and closes with a question and answer segment on hardware trade-offs (performance, power, area, accuracy, programmability) and examples of embedded and mobile deployments of compressed networks.

Before you watch

• Review how convolutional and recurrent networks are trained, since this lecture assumes familiarity with standard training loops and focuses on making them efficient.
• Basic familiarity with floating-point number representation is helpful for the quantization and mixed-precision sections.
• Knowing what an ASIC, FPGA, CPU, and GPU are in general terms will make the hardware sections easier to follow.

Check your understanding

1. Why does the lecture argue that memory access, not arithmetic, is the main energy cost in running deep neural networks?
2. How do pruning and weight sharing work together in Deep Compression, and what does each contribute to the final compression ratio?
3. What does the roofline model reveal about why a chip's real-world throughput on a given neural network can be far below its peak operations per second?
4. In mixed-precision training, why is FP32 still used for the weight update step even though FP16 is used elsewhere?
5. How does the EIE accelerator use sparsity to avoid unnecessary computation and memory traffic?