Skip to content

Neural Networks

NN

Module 1 · Neural Networks

Build workload intuition from tensors, backprop, CNNs, transformers, and training flow.

Artifact: model implementation note · Measure: loss, memory, FLOPs, throughput

Phase 3 — Artificial Intelligence (after Phase 1 §4 — C++ and Parallel Computing). Optional after Phase 2 if you prefer embedded first.

Goal: Build a concrete, ground-up understanding of what AI is, what artificial neural networks are, how they learn, and how to implement everything hands-on using tinygrad — the minimal ML framework that exposes every fundamental operation.

Start here if the math is unfamiliar: AI and Robot Learning Foundations provides worked gradients, tensor shapes, held-out evaluation, and a tested CPU robot-policy lab. Python and the foundation diagnostic are sufficient to begin; C++ and embedded hardware are useful later, not entry requirements for these calculations.

Code status: conceptual snippets below are exercises unless explicitly labeled otherwise. The MNIST adaptation is upstream-documented but has not been executed in this review. The foundation lab is the locally tested starting point. Record the framework version or source commit before adapting snippets; reviewed against primary documentation on 2026-10-18.

Related (separate topic): Edge AI — where models run on-device, latency/privacy tiers, train → optimize → deploy.


Deep Dive Subfolders

Subfolder Description
pytorch-and-micrograd/ Build autograd from scratch with micrograd, master PyTorch, then bridge to tinygrad — the essential prerequisite for deep tinygrad mastery

1. Training vs the edge

This guide focuses on training and inference math (usually on a workstation or cloud with enough RAM and FLOPs). On-device constraints—MCU vs SBC vs Jetson, latency, privacy, and the full train → quantize → deploy loop—live in Edge AI. You can read that track after the sections below, or skim it first for motivation.


2. What is Artificial Intelligence?

The Core Idea

Classical programming:

Rules + Data → Program → Output

Machine learning (the dominant AI approach today):

Data + Output (labels) → Learning Algorithm → Rules (model weights)

You don't write the rules. The algorithm learns the rules from examples.

Types of Machine Learning

Supervised Learning:
  Input: labeled pairs (image, "cat") (image, "dog")
  Learns: mapping from inputs to labels
  Examples: image classification, speech recognition, fraud detection

Unsupervised Learning:
  Input: unlabeled data
  Learns: hidden structure (clusters, patterns)
  Examples: customer segmentation, anomaly detection

Reinforcement Learning:
  Agent learns by trial and error in an environment
  Reward signal guides behavior
  Examples: game playing (AlphaGo), robot locomotion, autonomous driving

Where Neural Networks Fit

Neural networks are a family of parameterized models used in supervised, self-supervised, unsupervised, and reinforcement learning. Common applications include: - Images and video (CNNs) - Text and sequences (Transformers, RNNs) - Audio (CNNs + Transformers) - Tabular data (MLP) - Graphs (GNNs)

They can represent complex mappings. Whether training finds a useful mapping, and whether it generalizes to new conditions, depends on data, architecture, optimization, and evaluation. Representational capacity is not a guarantee of learning or reliable behavior.


3. What is a Neural Network?

Biological Inspiration

The brain contains ~86 billion neurons. Each neuron: - Receives electrical signals from other neurons via dendrites - Sums up the signals - If the sum exceeds a threshold, it fires (sends a signal to others) via the axon - The strength of connections between neurons is called synaptic weight

Learning = changing synaptic weights.

The Artificial Analogy

Biological Neuron          Artificial Neuron
─────────────────          ─────────────────
Dendrites                  Inputs  x₁, x₂, ..., xₙ
Synaptic weights           Weights w₁, w₂, ..., wₙ
Cell body summation        z = Σ(wᵢ · xᵢ) + b
Firing threshold           Activation function: a = f(z)
Axon output                Output a

A Network of Neurons

Neurons are organized into layers:

Input Layer     Hidden Layers      Output Layer
    x₁  ─────→  [neuron]  ─────→
    x₂  ─────→  [neuron]  ─────→  [neuron] → ŷ
    x₃  ─────→  [neuron]  ─────→
               [neuron]
  • Input layer: raw features (pixel values, sensor readings, etc.)
  • Hidden layers: learn intermediate representations
  • Output layer: final prediction (class probabilities, regression value)

The "deep" in deep learning = many hidden layers.


4. The Neuron: Building Block

Mathematical Definition

Given inputs x = [x₁, x₂, ..., xₙ]:

Step 1 — Linear combination (weighted sum):
  z = w₁x₁ + w₂x₂ + ... + wₙxₙ + b
    = w·x + b          (dot product notation)

Step 2 — Activation:
  a = f(z)             (apply nonlinearity)

Where: - w = weight vector (learned parameters) - b = bias (learned scalar offset) - f = activation function

Why the Bias?

Without bias, the decision boundary must pass through the origin. With bias:

z = wx + b
The bias shifts the activation, allowing the network to learn when to activate regardless of input magnitude. Think of it as the neuron's "default activation level."

Concrete Example

A toy neuron scores whether a simulated cart needs braking. The weights are invented for arithmetic practice, not learned or calibrated:

Inputs: x₁ = normalized speed = 1.5, x₂ = normalized obstacle proximity = 0.5
Weights: w₁ = 0.8, w₂ = 0.4
Bias: b = -1.0

z = 0.8 × 1.5 + 0.4 × 0.5 - 1.0 = 0.4

a = sigmoid(0.4) ≈ 0.5987

The sigmoid produces a number between 0 and 1. Interpreting it as a trustworthy probability requires calibration and held-out evidence; choosing a braking threshold also needs task-specific validation.


5. Activation Functions

Without activation functions, a stack of linear layers collapses to a single linear function. Nonlinearity is what gives neural networks their expressive power.

Sigmoid

σ(z) = 1 / (1 + e^(-z))

Range: (0, 1)
Use: binary classification output
Problem: vanishing gradients for large |z|
σ(z)
 1.0 |          ___________
 0.5 |        /
 0.0 |_______/
      -5   0   5     z

Tanh

tanh(z) = (e^z - e^(-z)) / (e^z + e^(-z))

Range: (-1, 1)
Use: hidden layers (historically), RNNs
Better than sigmoid: zero-centered
Problem: still has vanishing gradient

ReLU (Rectified Linear Unit)

ReLU(z) = max(0, z)

Range: [0, ∞)
Use: default for hidden layers in modern networks
Advantage: no vanishing gradient for z > 0, computationally fast
Problem: "dying ReLU" — neurons stuck at 0 if z always negative
ReLU(z)
  |          /
  |         /
  |        /
  |_______/
  -2  -1  0  1  2   z

Leaky ReLU

LeakyReLU(z) = z        if z > 0
              = 0.01·z   if z ≤ 0

Fixes dying ReLU by allowing small negative gradient

GELU (Gaussian Error Linear Unit)

GELU(z) = z · Φ(z), where Φ is the standard normal cumulative distribution

Used in: transformer feed-forward networks, including BERT and some GPT models
Smoothly weights inputs rather than applying a hard zero threshold

z × sigmoid(1.702z) is a particular approximation, not the definition. PyTorch also documents a tanh approximation; do not assume different approximations are numerically identical. See the GELU API.

Softmax (Output Layer for Multi-class)

softmax(z)ᵢ = e^(zᵢ) / Σⱼ e^(zⱼ)

Converts raw scores to probabilities that sum to 1
Use: multi-class classification output

Which to Use?

Layer Type Recommended Activation
Hidden (MLP/CNN) ReLU or GELU
Output (binary) Sigmoid
Output (multi) Softmax
Output (regression) None (linear)
RNN gates Tanh + Sigmoid

These output activations describe interpretation. During training, a loss that already includes the activation should receive logits: PyTorch CrossEntropyLoss takes unnormalized class scores, and BCEWithLogitsLoss includes the sigmoid calculation. Applying softmax or sigmoid twice changes the objective.


6. The Multi-Layer Perceptron (MLP)

Architecture

An MLP with 2 hidden layers:

Input        Hidden 1      Hidden 2      Output
(3 neurons)  (4 neurons)   (4 neurons)   (2 neurons)

  x₁ ─┐
  x₂ ─┼──→ [h₁¹]          [h₁²]
  x₃ ─┘    [h₂¹] ──────→  [h₂²] ──────→ [o₁]
            [h₃¹]          [h₃²]          [o₂]
            [h₄¹]          [h₄²]

Matrix Form

For a layer with n inputs and m neurons:

Z = X @ W + b

X: input matrix    shape [batch_size, n_inputs]
W: weight matrix   shape [n_inputs, n_neurons]
b: bias vector     shape [n_neurons]
Z: output          shape [batch_size, n_neurons]

Then: A = f(Z)   (apply activation element-wise)

This single matrix multiplication computes all neurons in a layer at once — highly parallelizable on GPU.

Parameter Count

Layer (n_in → n_out):
  Weights: n_in × n_out
  Biases:  n_out
  Total:   n_in × n_out + n_out

Example MLP: 784 → 256 → 128 → 10
  Layer 1: 784×256 + 256 = 201,216
  Layer 2: 256×128 + 128 = 32,896
  Layer 3: 128×10  + 10  = 1,290
  Total:   235,402 parameters

7. Forward Pass: Making a Prediction

The forward pass computes the output for a given input, layer by layer.

Step-by-Step (2-layer MLP)

# Pseudocode
def forward(x):
    # Layer 1
    z1 = x @ W1 + b1       # linear
    a1 = relu(z1)           # activation

    # Layer 2
    z2 = a1 @ W2 + b2      # linear
    a2 = relu(z2)           # activation

    # Output layer
    z3 = a2 @ W3 + b3      # linear
    output = softmax(z3)    # probabilities

    return output

What the Network Learns Layer by Layer

For image classification (MNIST digits):

Layer 1 (edges):    detects horizontal, vertical, diagonal edges
Layer 2 (shapes):   combines edges into curves, corners
Layer 3 (parts):    detects parts of digits (loops, lines)
Output (class):     combines parts into digit predictions

Each layer learns increasingly abstract representations of the input.


8. Loss Functions: Measuring Error

The loss (or cost) function measures how wrong the model's predictions are. Training = minimizing loss.

Mean Squared Error (MSE) — Regression

MSE = (1/N) Σᵢ (yᵢ - ŷᵢ)²

yᵢ  = true value
ŷᵢ  = predicted value
N   = number of samples

Good for: predicting continuous values (house price, temperature)

Binary Cross-Entropy — Binary Classification

BCE = -(1/N) Σᵢ [yᵢ log(ŷᵢ) + (1-yᵢ) log(1-ŷᵢ)]

yᵢ ∈ {0, 1}       true label
ŷᵢ ∈ (0, 1)       predicted probability (after sigmoid)

Intuition: penalizes confident wrong predictions heavily
  Predicted 0.99 when truth is 0 → very high loss
  Predicted 0.5  when truth is 0 → moderate loss

Categorical Cross-Entropy — Multi-class Classification

CCE = -(1/N) Σᵢ Σₖ yᵢₖ log(ŷᵢₖ)

yᵢₖ = 1 if sample i belongs to class k, else 0  (one-hot)
ŷᵢₖ = predicted probability for class k (after softmax)

Most common loss for image classification

Why Cross-Entropy?

Cross-entropy comes from information theory. Minimizing it is equivalent to maximum likelihood estimation — finding parameters that make the observed data most probable. It penalizes the model proportionally to how surprised it was by the correct answer.


9. Backpropagation: How Networks Learn

The Core Idea

Backpropagation = chain rule of calculus applied to compute gradients of the loss with respect to every weight in the network.

∂Loss/∂W = how much does the loss change when W changes slightly?

We want: decrease the loss
Strategy: move W in the direction that decreases Loss
Update:   W ← W - α · (∂Loss/∂W)

Chain Rule Foundation

For a composed function f(g(x)):

df/dx = (df/dg) · (dg/dx)

In a neural network, the loss flows through many composed functions:

Loss = L(softmax(W₃ · relu(W₂ · relu(W₁ · x + b₁) + b₂) + b₃))

Backprop applies chain rule backwards from output to input.

Computational Graph

Think of the network as a graph of operations:

x → [W₁ matmul] → z₁ → [relu] → a₁ → [W₂ matmul] → z₂ → [loss] → L

Forward pass: compute and cache every intermediate value. Backward pass: compute gradient at each node using cached values + chain rule.

Step-by-Step Backward Pass (1 layer example)

Forward:
  z = x·w + b
  a = relu(z)
  L = MSE(a, y)

Backward (chain rule):
  dL/da = 2(a - y)/N                    (MSE gradient)
  dL/dz = dL/da · d(relu)/dz           (relu gradient: 1 if z>0 else 0)
  dL/dw = dL/dz · x                    (matmul gradient)
  dL/db = dL/dz · 1                    (bias gradient)
  dL/dx = dL/dz · w                    (input gradient, for prev layer)

Automatic Differentiation (Autograd)

Modern frameworks (including tinygrad) implement autograd: they build the computational graph during the forward pass, then automatically compute all gradients during .backward().

Usually you define the forward pass and let autograd compute derivatives. Implement a small backward pass and finite-difference check once so you can debug custom operations and incorrect gradients.

# tinygrad autograd example
from tinygrad import Tensor

x = Tensor([2.0], requires_grad=True)
w = Tensor([3.0], requires_grad=True)
b = Tensor([1.0], requires_grad=True)

z = x * w + b          # forward: builds graph
loss = z.pow(2).mean() # loss computation

loss.backward()         # backward: compute all gradients

print(w.grad)  # dL/dw computed automatically

Vanishing and Exploding Gradients

Vanishing gradients: in deep networks, gradients become extremely small as they propagate back through many layers. - Cause: sigmoid/tanh saturate (derivative ≈ 0) - Fix: ReLU, residual connections (ResNets), batch normalization

Exploding gradients: gradients grow exponentially. - Cause: large weights × many layers - Fix: gradient clipping, weight initialization schemes


10. Gradient Descent and Optimizers

Gradient Descent (Base Algorithm)

For each training step:
  1. Compute loss on a batch of data
  2. Compute gradients via backprop
  3. Update every parameter:
     W ← W - α · ∂Loss/∂W

α = learning rate (hyperparameter, typically 1e-3 to 1e-4)

Batch Variants

Variant Batch Size Pros Cons
Batch GD Full dataset Stable gradients Slow, memory-intensive
Stochastic GD (SGD) 1 sample Fast updates Very noisy
Mini-batch GD 32–256 Balance of both Industry standard

SGD with Momentum

vₜ = β·vₜ₋₁ + (1-β)·∇L     (momentum term, β ≈ 0.9)
W ← W - α·vₜ

Intuition: gradient as a ball rolling downhill, accumulates velocity
Benefit: faster convergence, escapes shallow local minima

Adam (Adaptive Moment Estimation)

The most widely used optimizer:

mₜ = β₁·mₜ₋₁ + (1-β₁)·∇L          (1st moment: mean of gradients)
vₜ = β₂·vₜ₋₁ + (1-β₂)·(∇L)²       (2nd raw moment: mean squared gradients)

m̂ₜ = mₜ/(1-β₁ᵗ)                   (bias correction)
v̂ₜ = vₜ/(1-β₂ᵗ)

W ← W - α · m̂ₜ / (√v̂ₜ + ε)

Defaults: β₁=0.9, β₂=0.999, ε=1e-8, α=1e-3

Adam adapts the learning rate per parameter based on historical gradients. Works well out of the box for most tasks.

Learning Rate Scheduling

Constant LR:       α = 0.001 throughout
Step decay:        α = α₀ × 0.1 every 10 epochs
Cosine annealing:  α follows a cosine curve
Warmup + decay:    Start low, increase, then decrease (Transformers)

11. The Training Loop

Complete Training Loop

# Framework-neutral pseudocode using PyTorch-style train/eval names.
# tinygrad uses its own documented training context; do not mix APIs.

model = MyNetwork()
optimizer = Adam(model.parameters(), lr=1e-3)
loss_fn = CrossEntropyLoss()

for epoch in range(num_epochs):
    # ── Training phase ──────────────────────────────
    model.train()
    for batch_x, batch_y in train_loader:

        # 1. Forward pass
        predictions = model(batch_x)

        # 2. Compute loss
        loss = loss_fn(predictions, batch_y)

        # 3. Zero gradients (clear previous step's gradients)
        optimizer.zero_grad()

        # 4. Backward pass (compute gradients)
        loss.backward()

        # 5. Update weights
        optimizer.step()

    # ── Validation phase ─────────────────────────────
    model.eval()
    # evaluate must also disable gradient recording; eval() alone does not.
    val_loss, val_acc = evaluate(model, val_loader)
    print(f"Epoch {epoch}: loss={val_loss:.4f}, acc={val_acc:.2%}")

Key Concepts in the Loop

Epoch: one complete pass through the entire training dataset.

Batch: a subset of the dataset processed together (e.g., 32 images). Enables: - GPU parallelism - Gradient averaging (more stable than single samples) - Fitting large datasets in memory

Overfitting: model memorizes training data, fails on new data.

Training loss:   ↓ ↓ ↓ ↓ ↓ 0.01
Validation loss: ↓ ↓ ↑ ↑ ↑ 0.45   ← overfitting after this point

Underfitting: model too simple, fails on both training and validation.

Training loss:   0.40 (stays high)
Validation loss: 0.42 (also high)

Hyperparameters vs Parameters

Parameters (learned by training):
  - Weights W
  - Biases b

Hyperparameters (set by you before training):
  - Learning rate α
  - Batch size
  - Number of layers
  - Number of neurons per layer
  - Number of epochs
  - Dropout rate
  - Weight decay

12. Convolutional Neural Networks (CNNs)

Why Not Just Use MLP for Images?

A 224×224 RGB image = 224×224×3 = 150,528 inputs. MLP first hidden layer with 1024 neurons = 150,528 × 1024 = 154 million parameters for one layer.

Problems: - Doesn't capture spatial structure (neighboring pixels matter together) - No weight sharing (same edge detector at every location) - Too many parameters → overfitting

The Convolution Operation

A filter (kernel) slides over the input, computing a weighted sum at each position:

Input (5×5):          Filter (3×3):      Output (3×3):
 1  2  3  4  5         1  0  1            ?  ?  ?
 5  6  7  8  9         0  1  0            ?  ?  ?
 1  2  3  4  5         1  0  1            ?  ?  ?
 5  6  7  8  9
 1  2  3  4  5

At top-left position:
  (1×1)+(2×0)+(3×1) + (5×0)+(6×1)+(7×0) + (1×1)+(2×0)+(3×1)
  = 1+3+6+1+3 = 14

The filter is a learned pattern detector. The network learns filters that detect edges, textures, shapes automatically.

CNN Architecture Components

Input Image
    ↓
Conv Layer (learns feature maps)
    ↓
Activation (ReLU)
    ↓
Pooling (downsample, reduce size)
    ↓
... (repeat)
    ↓
Flatten
    ↓
FC (Fully Connected) Layers
    ↓
Output (Softmax)

Pooling

Max pooling reduces spatial dimensions, retaining the most prominent features:

Input (4×4):       Max Pool 2×2, stride 2, no padding:
 1  3  2  4
 5  6  7  8    →     6  8
 3  1  4  2          8  6
 7  8  5  6

CNN vs MLP for Images

MLP:
  - Every pixel connects to every neuron
  - No spatial structure preserved
  - Millions of redundant parameters

CNN:
  - Local connectivity (filter slides over image)
  - Weight sharing (same filter at all positions)
  - Translation equivariant in ideal stride-1 convolution away from boundaries
  - Pooling/aggregation can add some translation tolerance; invariance is not automatic
  - Far fewer parameters

Classic CNN Architectures

Architecture Design idea to inspect Your comparison artifact
LeNet / AlexNet Convolution, pooling, and dense heads Input/output shapes and parameter counts
VGG Repeated small convolutions Activation and weight memory by layer
ResNet Residual connections Shape matching at each addition
MobileNetV2 Depthwise convolutions and inverted residual blocks Per-operator time on the target device
EfficientNet Joint scaling choices Quality/latency/memory at a named input size

Accuracy belongs to a checkpoint, dataset split, preprocessing, and evaluation procedure. Compare measured models under the same protocol; a model-family name and an unqualified top-5 percentage are not a reproducible benchmark.


13. Regularization and Generalization

The Bias-Variance Tradeoff

Total Error = Bias² + Variance + Irreducible Noise

High Bias (underfitting):   model too simple, misses patterns
High Variance (overfitting): model too complex, memorizes noise

Dropout

Randomly zero out neurons during training:

During training:
  For each neuron, with probability p, set output to 0
  Scale remaining neurons by 1/(1-p)

# During inference:
  No dropout (all neurons active)

Effect: Forces network to learn redundant representations
        Acts as ensemble of many sub-networks
Typical p: 0.2–0.5

L2 Regularization and Weight Decay

Add a penalty to the loss for large weights:

L_total = L_task + λ · Σ w²

λ = regularization strength (e.g., 1e-4)

Effect: Keeps weights small, reduces overfitting
Equivalent to Gaussian prior on weights (Bayesian view)

For plain SGD, an L2 gradient term can be expressed as a weight-decay update with the appropriate coefficient. With adaptive optimizers such as Adam, this equivalence generally fails. AdamW applies decay separately; see Decoupled Weight Decay Regularization.

Batch Normalization

Normalize the inputs to each layer:

For a mini-batch of activations z:
  μ = mean(z)
  σ² = var(z)
  z_norm = (z - μ) / √(σ² + ε)
  output = γ · z_norm + β        (γ, β are learned)

Benefits:
  - Faster training (higher learning rates)
  - Less sensitive to weight initialization
  - Mild regularization effect
  - Changes optimization behavior; the mechanism is not fully explained by
    the original “internal covariate shift” intuition

See How Does Batch Normalization Help Optimization? for evidence against treating that original explanation as settled. Training and evaluation also use different statistics in the usual BatchNorm configuration; preserve the running statistics with the weights.

Early Stopping

Monitor validation loss during training
Stop training when validation loss stops improving

Epoch: 1  train_loss=2.3  val_loss=2.3
Epoch: 5  train_loss=1.2  val_loss=1.3
Epoch:10  train_loss=0.5  val_loss=0.8   ← save checkpoint here
Epoch:15  train_loss=0.2  val_loss=1.1   ← overfitting, stop
Epoch:20  train_loss=0.1  val_loss=1.4

Data Augmentation

Artificially expand dataset by transforming existing samples:

Image augmentations:
  - Horizontal/vertical flip
  - Random crop and resize
  - Color jitter (brightness, contrast, saturation)
  - Random rotation
  - Gaussian noise

Effect: Model sees more variety → better generalization
Edge AI benefit: reduces need for large datasets

14. Hands-On with tinygrad

Why tinygrad?

tinygrad is a compact tensor and compiler framework whose source is useful for tracing operations through scheduling and code generation. Its size, layout, and supported backends change; inspect the pinned revision rather than relying on a fixed line count.

  • No magic — you can read and understand the entire source
  • Teaches you exactly what happens inside a neural network
  • Compiles to CPU, GPU, CUDA, Metal, WebGPU
  • Used in production at comma.ai (autonomous driving)

tinygrad Core Concepts

from tinygrad.tensor import Tensor
import numpy as np

# Tensor creation
x = Tensor([[1.0, 2.0, 3.0]])           # shape [1, 3]
w = Tensor.kaiming_uniform(3, 4, requires_grad=True)  # shape [3, 4]

# Operations (lazy by default)
z = x.matmul(w)                         # shape [1, 4]
a = z.relu()

# Realize (execute computation)
result = a.numpy()                       # triggers actual computation

# Gradient computation
loss = a.sum()
loss.backward()
print(w.grad.numpy())                   # ∂loss/∂w

Implementing a Neuron from Scratch

from tinygrad.tensor import Tensor

class Neuron:
    def __init__(self, n_inputs):
        # Initialize weights with small random values
        self.w = Tensor.randn(n_inputs, 1) * 0.01
        self.b = Tensor.zeros(1)

    def __call__(self, x):
        z = x.matmul(self.w) + self.b
        return z.relu()

    def parameters(self):
        return [self.w, self.b]

# Test
neuron = Neuron(3)
x = Tensor([[0.5, 1.2, -0.3]])
output = neuron(x)
print(output.numpy())  # shape [1, 1]

Implementing a Layer

class Linear:
    def __init__(self, n_in, n_out):
        # Kaiming initialization for ReLU
        self.w = Tensor.kaiming_uniform(n_in, n_out)
        self.b = Tensor.zeros(n_out)

    def __call__(self, x):
        return x.matmul(self.w) + self.b

    def parameters(self):
        return [self.w, self.b]

Implementing an MLP

class MLP:
    def __init__(self, layers):
        # layers = [784, 256, 128, 10]
        self.linears = [
            Linear(layers[i], layers[i+1])
            for i in range(len(layers)-1)
        ]

    def __call__(self, x):
        for i, layer in enumerate(self.linears[:-1]):
            x = layer(x).relu()          # hidden layers: ReLU
        x = self.linears[-1](x)          # output layer: no activation
        return x                         # logits; loss applies log-softmax

    def parameters(self):
        params = []
        for layer in self.linears:
            params.extend(layer.parameters())
        return params

Training on MNIST

Upstream-documented adaptation, not locally executed. The current tinygrad quickstart uses a training context and a loss on logits. The example below assumes the MLP class above returns logits. Record a tinygrad source commit/package version, NumPy version, dataset checksums, and device before using it as a reproducible experiment. The API review date is 2026-10-18; that date is not a software pin.

from tinygrad import Tensor, Context
from tinygrad.nn.optim import Adam
from tinygrad.helpers import fetch
import gzip
import numpy as np

rng = np.random.default_rng(7)
Tensor.manual_seed(7)

def load_mnist():
    base = "https://storage.googleapis.com/cvdf-datasets/mnist/"
    def read(name, offset):
        with gzip.open(fetch(base + name), "rb") as stream:
            return np.frombuffer(stream.read(), np.uint8, offset=offset)
    train_x = read("train-images-idx3-ubyte.gz", 16).reshape(-1, 784)
    train_y = read("train-labels-idx1-ubyte.gz", 8)
    test_x = read("t10k-images-idx3-ubyte.gz", 16).reshape(-1, 784)
    test_y = read("t10k-labels-idx1-ubyte.gz", 8)
    return train_x.astype(np.float32) / 255, train_y, test_x.astype(np.float32) / 255, test_y

all_x, all_y, test_x, test_y = load_mnist()
order = rng.permutation(len(all_x))
train_ids, val_ids = order[5000:], order[:5000]
train_x, train_y = all_x[train_ids], all_y[train_ids]
val_x, val_y = all_x[val_ids], all_y[val_ids]
model = MLP([784, 256, 128, 10])
optimizer = Adam(model.parameters(), lr=1e-3)
BATCH, EPOCHS = 64, 10  # fixed budget; use validation if changing it

def accuracy(xs, ys):
    correct = 0
    with Context(TRAINING=0):
        for i in range(0, len(xs), BATCH):
            predictions = model(Tensor(xs[i:i+BATCH])).numpy().argmax(axis=1)
            correct += int((predictions == ys[i:i+BATCH]).sum())
    return correct / len(xs)

for epoch in range(EPOCHS):
    order = rng.permutation(len(train_x))
    loss_sum, examples = 0.0, 0
    with Context(TRAINING=1):
        for i in range(0, len(order), BATCH):
            ids = order[i:i+BATCH]
            xb = Tensor(train_x[ids])
            yb = Tensor(train_y[ids].astype(np.int32))
            logits = model(xb)
            loss = logits.sparse_categorical_crossentropy(yb)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            loss_sum += loss.item() * len(ids)
            examples += len(ids)
    print(f"epoch={epoch+1} loss={loss_sum/examples:.4f} validation_accuracy={accuracy(val_x, val_y):.4f}")

# Once the candidate is fixed, evaluate the untouched test partition.
print(f"final_test_accuracy={accuracy(test_x, test_y):.4f}")

A complete lab also saves the parameters, split IDs, exact environment, and raw metrics. If you tune after reading the test score, report that set as development evidence and use a fresh evaluation for a final generalization claim. The CPU foundation lab is the tested route when this framework or network access is unavailable.

Understanding tinygrad's Autograd

# tinygrad builds a computation graph lazily
# Every operation on a Tensor records itself

x = Tensor([2.0], requires_grad=True)
y = x * x       # records: mul(x, x)
z = y + x       # records: add(y, x)
loss = z.sum()

# .backward() traverses graph in reverse (topological sort)
# applying chain rule at each node
loss.backward()

print(x.grad)   # d(loss)/d(x) = d(x²+x)/dx = 2x+1 = 5.0

Implementing a CNN in tinygrad

from tinygrad.tensor import Tensor
from tinygrad.nn import Conv2d, BatchNorm2d
from tinygrad.nn.optim import Adam

class SimpleCNN:
    def __init__(self):
        # Conv layers
        self.c1 = Conv2d(1, 32, 3, padding=1)   # 1 channel in, 32 out, 3×3 kernel
        self.c2 = Conv2d(32, 64, 3, padding=1)

        # Fully connected
        self.fc1 = Linear(64 * 7 * 7, 128)
        self.fc2 = Linear(128, 10)

    def __call__(self, x):
        # x shape: [batch, 1, 28, 28]
        x = self.c1(x).relu().max_pool2d()       # → [batch, 32, 14, 14]
        x = self.c2(x).relu().max_pool2d()       # → [batch, 64, 7, 7]
        x = x.reshape(x.shape[0], -1)            # flatten → [batch, 3136]
        x = self.fc1(x).relu()
        x = self.fc2(x)
        return x  # logits

    def parameters(self):
        params = []
        for layer in [self.c1, self.c2, self.fc1, self.fc2]:
            params.extend(layer.parameters() if hasattr(layer, 'parameters') else [layer.weight, layer.bias])
        return params

Exploring tinygrad Internals

Use the upstream layout guide with your pinned source revision. The current guide describes scheduling, code-generation optimization, rendering, and execution in schedule/, codegen/opt/, codegen/, renderer/, and engine/. Trace one small matmul through those stages and retain the generated program. Old tutorials referring to lazybuffer.py or runtime/ops_cpu.py describe earlier layouts; do not assume those paths exist in your checkout.


15. Projects

Project 1: Neuron from Scratch (No Framework)

Implement a single neuron in pure Python/NumPy. Manually compute gradients and verify with finite differences.

Goal: understand forward pass, loss, gradient by hand
Dataset: XOR problem (4 samples)
Deliverable: working neuron with manual backprop

Project 2: MLP for MNIST with tinygrad

Train a fully connected network on MNIST digit classification.

Goal: achieve >97% test accuracy
Architecture: 784 → 256 → 128 → 10
Deliverable: training script + loss/accuracy curves

Project 3: CNN for MNIST/CIFAR-10 with tinygrad

Replace MLP with a convolutional network.

Goal: understand convolution, pooling, feature maps
Dataset: MNIST (>99%) or CIFAR-10 (>85%)
Deliverable: CNN training script, visualize learned filters

Project 4: Implement Adam from Scratch

Implement the Adam optimizer manually in tinygrad (without using tinygrad's built-in Adam).

Goal: understand optimizer internals
Deliverable: custom Adam class matching tinygrad's results

Project 5: Read tinygrad's Backprop

Trace through tinygrad source code for a simple operation (e.g., x * x) and document exactly what happens during backward().

Goal: understand autograd mechanics
Deliverable: annotated source code walkthrough
Files: tensor.py, ops.py

Project 6: MNIST on Jetson (Edge Deployment)

Train on desktop, export model weights, run inference on Jetson Nano.

Goal: complete edge AI pipeline
Steps:
  1. Train CNN on desktop
  2. Export weights as numpy arrays
  3. Load weights in tinygrad on Jetson
  4. Measure inference latency: CPU vs GPU
  5. Quantize to INT8, compare accuracy/speed

16. Resources

Foundational Theory

  • 3Blue1Brown — Neural Networks (YouTube series): Best visual introduction to how neural networks work. Watch all 4 episodes before writing code.
  • "But what is a neural network?"
  • "Gradient descent, how neural networks learn"
  • "What is backpropagation really doing?"
  • "Backpropagation calculus"

  • Andrej Karpathy — micrograd (GitHub + YouTube): Build autograd from scratch in ~150 lines. Essential for understanding backprop deeply. Then compare with tinygrad's implementation.

  • https://github.com/karpathy/micrograd

  • CS231n: Convolutional Neural Networks for Visual Recognition (Stanford): The definitive CNN course. Lecture notes are excellent even without watching videos.

  • https://cs231n.github.io/

  • The Deep Learning Book (Goodfellow, Bengio, Courville): Free online. Chapters 6-9 cover MLP, backprop, regularization, CNNs rigorously.

  • https://www.deeplearningbook.org/

tinygrad-Specific

  • tinygrad source code: The best tinygrad documentation is the source itself.
  • tensor.py for ops and autograd
  • examples/ for training scripts
  • test/ for understanding expected behavior

  • tinygrad MNIST example: tinygrad/examples/mnist.py — canonical starting point

  • tinygrad CNN example: tinygrad/examples/efficientnet.py

Edge AI Context

  • TinyML book (Pete Warden, Daniel Situnayake): Running ML on microcontrollers. Chapters 1-3 give strong context for why edge AI design choices matter.

  • AI at the Edge (Daniel Situnayake, Jenny Plunkett): End-to-end edge AI system design.

Math Prerequisites

  • Linear Algebra: 3Blue1Brown "Essence of Linear Algebra" — especially matrices as transformations
  • Calculus: Chain rule, partial derivatives — Khan Academy calculus or 3Blue1Brown "Essence of Calculus"
  • Statistics: Probability, distributions — needed for loss functions and regularization

Practice Datasets

Dataset Task Input First comparison
MNIST Digit classification 28×28 grayscale Majority class, linear model, MLP
Fashion-MNIST Clothing classification 28×28 grayscale Same fixed model and split protocol
CIFAR-10 Object classification 32×32 RGB Small CNN with recorded augmentation
Iris Flower classification Four tabular features Simple classifier with stratified validation

Use each dataset's documented split, license, and checksum. Establish a baseline with your own exact training and evaluation recipe; this table makes no promised accuracy claim. For robot data, use whole episodes and task-relevant groups as explained in evaluation foundations.


Quick Reference: Key Equations

Neuron forward:
  z = Wx + b
  a = f(z)

MSE Loss:
  L = (1/N) Σ(y - ŷ)²

Cross-entropy Loss:
  L = -(1/N) Σ y·log(ŷ)

Gradient descent:
  W ← W - α·∂L/∂W

Adam update:
  m = β₁m + (1-β₁)g
  v = β₂v + (1-β₂)g²
  W ← W - α·m̂/(√v̂ + ε)

Convolution output size:
  out = floor((in + 2p - k) / s) + 1
  in=input, p=padding, k=kernel, s=stride

Deep Dive: Subfolders

PyTorch + micrograd → tinygrad

Before diving deep into tinygrad internals or porting models, you need a rock-solid foundation in how autograd actually works. This subfolder provides exactly that:

  • Part 1 — micrograd: Reimplement Karpathy's scalar autograd engine from scratch. Every +, *, tanh, exp carries a _backward closure. You'll see chain rule as running code, not math notation.
  • Part 2 — PyTorch: Aligned with the OpenCV PyTorch Bootcamp curriculum. Tensors, autograd, nn.Module, CNNs, transfer learning, object detection heads — with the same mental model as micrograd.
  • Part 3 — Bridge to tinygrad: Side-by-side comparison of the same MLP in all three frameworks. What PyTorch hides (lazy evaluation, kernel fusion) that tinygrad exposes via DEBUG=4.

Why this matters for tinygrad mastery:

micrograd  →  you understand every gradient operation
PyTorch    →  you understand the production API and patterns
tinygrad   →  you see what frameworks hide, control the scheduler


Next (Phase 4 Track B): ML and AI — quantization, pruning, TensorRT, TFLite

Study method: use the shared lesson cycle and completion rubric with this course's existing labs. Predict first, test one variation, retain a failure case, and revisit the self-check after two days and one week.

Carry this course into the physical-AI system

Start from: A small labeled dataset and a two-layer network.

Build and challenge: Derive one forward pass and one gradient by hand. Compare finite differences away from nondifferentiable points, then deliberately transpose one weight to explain the shape failure.

Pass forward: A numerical reference, tensor-shape table, and measured training/inference distinction. Continue with the connected course and keep the artifact in the Physical AI Design Studio.