2025-10-19 | Shannon

Goal: Shannon 1948 Discrete Noiseless Channel

Summary: Multi-disciplinary approach to Interpretability, viewing models through the lens of Entropy

Work sessions

In	Out
19:00	23:00

Why Information Theory

Shannon 1948 is a seminal text on the way scientists/engineers view how computer's transmit, store, and produce information.

While written 77 years ago, concepts like Entropy (in fact, LLMs are trained on Cross-Entropy loss functions!), Mutual Information, and Noise perhaps may lead to greater insight which emerge from model training pressures.

Nicely intersected with a reading for my Phonology Course