The Illustrated Vision Transformer

Dummy demonstration post — structure only

Note

Placeholder content demonstrating the v1 authoring surface (design.md §4.1): LaTeX math, static syntax-highlighted code blocks, figures, and callouts.

From words to patches

A Vision Transformer (ViT) treats an image exactly the way a language model treats a sentence: as a sequence of tokens. An image \(x \in \mathbb{R}^{H \times W \times C}\) is split into \(N\) flattened patches \(x_p \in \mathbb{R}^{N \times (P^2 C)}\), each linearly projected into the model dimension:

\[ z_0 = [\,x_\text{class};\; x_p^1 E;\; x_p^2 E;\; \dots;\; x_p^N E\,] + E_{pos} \]

where \(E \in \mathbb{R}^{(P^2 C) \times D}\) is the patch embedding and \(E_{pos}\) the learned position embedding.

Attention, briefly

Each head computes scaled dot-product attention:

\[ \mathrm{Attention}(Q, K, V) = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right) V \]

# static code block — not executed at render time (design.md §4.1)
import numpy as np

def attention(q, k, v):
    scores = q @ k.T / np.sqrt(k.shape[-1])
    weights = np.exp(scores - scores.max(-1, keepdims=True))
    weights /= weights.sum(-1, keepdims=True)
    return weights @ v

What the patches see

Tip

Figures live next to the post (figures/…) and are re-encoded at build time to strip EXIF metadata (design.md §4.3).

(figure placeholder — a real post embeds attention-map visualizations here)

Further reading

• Dosovitskiy et al., An Image is Worth 16x16 Words (2020)
• The annotated-transformer genre this site follows