Skip to content

2026-05-27 | Maxwell

Goal: Prepare for Syntax of LLMs Project kickoff

Summary: Writeup official proposal and research questions for investigating the syntax of LLMs

Work sessions

In Out Task
08:00 09:00 Learn Japanese Adverb, Adjective, and Relative Clause Structure
15:05 17:00 Change CFG to explain different verbs

Goals

  1. [P0] Design grammar to contain relative clauses, adjectives, and adverbs

  2. [P0] Train existing Vaswani et al. Transformer on (a) Depth {0,1,2} with holdout depth {3} and (b) uniformly sample depths {0,1,2,3}

  3. [P1] Train Toy GPT-2 Small style architecture on (a) and (b) above

Others

  1. Complete required REU Survey

Meetings

  1. None

Summary

  1. ✅ Designed grammar to include Bare NPs (e.g. “John”), Transitive, Intransitive, and CP specific verbs

  2. 🟡 [in progress] [P0] Train existing Vaswani et al. Transformer on (a) Depth {0,1,2} with holdout depth {3} and (b) uniformly sample depths {0,1,2,3}

  3. 🟡 [not started] [P0] Train Toy GPT-2 Small style architecture on (a) and (b) above

Additional

  1. Added TODO: understand how RoPE works

  2. Added zsh and oh-my-zsh plugin to user in cluster