large language models, often implemented as autoregressive transformers models.

GPTs and friends

Most variants of LLMs are decoder-only (Radford et al., 2019)

Have “capabilities” to understand natural language.

Exhibits emergent behaviour of intelligence, but probably not AGI due to observer-expectancy effect.

One way or another is a form of behaviourism, through reinforcement learning. It is being “told” what is good or bad, and thus act accordingly towards the users. However, this induces confirmation bias where one aligns and contains his/her prejudices towards the problem.

• url: thoughts/mechanistic-interpretability

mechanistic interpretability

whirlwind tour, initial exploration, glossary

The subfield of alignment that delves into reverse engineering of a neural network, especially LLMs

To attack the curse of dimensionality, the question remains: how do we hope to understand a function over such a large space, without an exponential amount of time? ²

Topics:

• sparse autoencoder
• sparse crosscoders
• Attribution parameter decomposition

open problems

Sharkey et al. (2025)

• differentiate between “reverse engineering” versus “concept-based”
  • reverse engineer:
    • decomposition → hypotheses → validation
      • Decomposition via dimensionality reduction
  • drawbacks with SDL:
    • SDL reconstruction error are way too high (Rajamanoharan et al., 2024, p. see section 2.3)
    • SDL assumes linear representation hypothesis against non-linear feature space.
    • SDL leaves feature geometry unexplained

inference

Application in the wild: Goodfire and Transluce

How we would do inference with SAE?

Quick 🧵 and some of quick introspection into how they might run inference https://t.co/1N3JrxFHSp

— aaron (@aarnphm_) 25 septembre 2024

idea: treat SAEs as a logit bias, similar to guided decoding

steering

refers to the process of manually modifying certain activations and hidden state of the neural net to influence its outputs

For example, the following is a toy example of how a decoder-only transformers (i.e: GPT-2) generate text given the prompt “The weather in California is”

flowchart LR
  A[The weather in California is] --> B[H0] --> D[H1] --> E[H2] --> C[... hot]

To steer to model, we modify $H_2$ layers with certain features amplifier with scale 20 (called it $H_{3}$)³

flowchart LR
  A[The weather in California is] --> B[H0] --> D[H1] --> E[H3] --> C[... cold]

One usually use techniques such as sparse autoencoders to decompose model activations into a set of interpretable features.

For feature ablation, we observe that manipulation of features activation can be strengthened or weakened to directly influence the model’s outputs

example: Panickssery et al. (2024) uses contrastive activation additions to steer Llama 2

contrastive activation additions

intuition: using a contrast pair for steering vector additions at certain activations layers

Uses mean difference which produce difference vector similar to PCA:

Given a dataset $\mathcal{D}$ of prompt $p$ with positive completion $c_p$ and negative completion $c_n$, we calculate mean-difference $v_\text{MD}$ at layer $L$ as follow:

$$v_\text{MD} = \frac{1}{\mid \mathcal{D} \mid} \sum_{p,c_p,c_n \in \mathcal{D}} a_L(p,c_p) - a_L(p, c_n)$$

implication

by steering existing learned representations of behaviors, CAA results in better out-of-distribution generalization than basic supervised finetuning of the entire model.

superposition hypothesis

tl/dr

phenomena when a neural network represents more than $n$ features in a $n$-dimensional space

Linear representation of neurons can represent more features than dimensions. As sparsity increases, model use superposition to represent more features than dimensions.

neural networks “want to represent more features than they have neurons”.

When features are sparsed, superposition allows compression beyond what linear model can do, at a cost of interference that requires non-linear filtering.

reasoning: “noisy simulation”, where small neural networks exploit feature sparsity and properties of high-dimensional spaces to approximately simulate much larger much sparser neural networks

In a sense, superposition is a form of lossy compression

importance

• sparsity: how frequently is it in the input?

• importance: how useful is it for lowering loss?

over-complete basis

reasoning for the set of $n$ directions ⁴

features

A property of an input to the model

When we talk about features (Elhage et al., 2022, p. see “Empirical Phenomena”), the theory building around several observed empirical phenomena:

1. Word Embeddings: have direction which corresponding to semantic properties (Mikolov et al., 2013). For example:

   V(king) - V(man) = V(monarch)

2. Latent space: similar vector arithmetics and interpretable directions have also been found in generative adversarial network.

We can define features as properties of inputs which a sufficiently large neural network will reliably dedicate a neuron to represent (Elhage et al., 2022, p. see “Features as Direction”)

ablation

refers to the process of removing a subset of a model’s parameters to evaluate its predictions outcome.

idea: deletes one activation of the network to see how performance on a task changes.

• zero ablation or pruning: Deletion by setting activations to zero
• mean ablation: Deletion by setting activations to the mean of the dataset
• random ablation or resampling

residual stream

intuition: we can think of residual as highway networks, in a sense portrays linearity of the network ⁵

residual stream $x_{0}$ has dimension $\mathit{(C,E)}$ where

• $\mathit{C}$: the number of tokens in context windows and
• $\mathit{E}$: embedding dimension.

Attention mechanism $\mathit{H}$ process given residual stream $x_{0}$ as the result is added back to $x_{1}$:

$$x_{1} = \mathit{H}{(x_{0})} + x_{0}$$

grokking

See also: writeup, code, circuit threads

A phenomena discovered by (Power et al., 2022) where small algorithmic tasks like modular addition will initially memorise training data, but after a long time ti will suddenly learn to generalise to unseen data

empirical claims

related to phase change

Lien vers l'original