2026-03-23 | Transformer From Scratch

Goal: Transformer from scratch: Implement Toy IOI Model

Summary: Progress: Finished Transformer Architecture

Work sessions

In	Out
06:30	07:30

Goals

AutoInterp

Copying these over from yesterday:

1. Implement Toy IOI Transformer from scratch
2. Use Claude to verify that my Pytorch Transformer implementation is correct
3. Create Dataset of IOI Sentences
4. Train Model and verify model generalization
5. Understand how layer norms can be folded into the surrounding parts of a model

Linguistics

1. Hodge Laplacian, understand how to derive/hand calculate

2. CCG for Dyck Language