You can now capture per-layer activation vectors from llama-server during inference, train sparse autoencoders on them, discover which internal features correspond to specific behaviors (sycophancy, hedging, creativity, etc.), and extract those features as GGUF control vectors for real-time steering.

**What this is:**

A C++ patch to llama-server that adds `/activations` endpoints, plus a Python pipeline for the full SAE workflow. The patch is \\~400 lines across 5 files and adds:

* `GET /activations`: query per-layer mean activations (with top-K filtering)

* `POST /activations`: enable/disable capture

* `POST /activations/collect`: stream full per-token vectors to a binary file for offline training

**What you can do with it:**

1. **Monitor activations live**: see which features fire strongest during a conversation
2. **Collect training data**: stream per-token activation vectors to disk while running inference
3. **Train a sparse autoencoder**: decompose activations into \\~16K interpretable features (takes about 40 seconds on an RTX 3090)
4. **Discover behavioral features**: define phrase clusters ("sycophantic phrases", "hedging phrases", etc.) and find which features are unique to each behavior
5. **Extract control vectors**: turn discovered features into GGUF files you can load with `--control-vector-scaled`
6. **Steer in real time**: suppress sycophancy, amplify creativity, whatever you want, at the feature level

**How it works technically:**

The patch hooks into llama.cpp's existing `cb\_eval` callback to intercept `l\_out` tensors (layer outputs) during the forward pass. GPU→CPU copy via `ggml\_backend\_tensor\_get()`, stored in a mutex-protected global struct. The binary collection format is dead simple: 16-byte header + float32 arrays, directly readable with numpy.

The SAE pipeline is standard: collect activations → train sparse autoencoder → probe features with behavioral phrase clusters → extract feature directions as control vectors. The interesting part is the inter-cluster differential scoring: instead of just finding "features that fire on sycophantic text," it finds features that fire *significantly more* on sycophantic text than on any other cluster, so you get specific behavioral features rather than generic language features.

**PR + repo:**

* llama.cpp PR: https://github.com/ggml-org/llama.cpp/pull/20785

* Companion repo with the full SAE pipeline, guide, and example clusters: https://github.com/hrhdegenetrix/llama-sae-feature-interpretability

The companion repo has a quickstart script, example behavioral cluster definitions, and a comprehensive guide covering the full workflow.

**Notes:**

* MoE models are *extremely* sensitive to control vector scales. Dense models (Qwen3-8B, 4096 embd) handle scales of 0.15-0.6 fine. Qwen3.5-35B-A3B MoE (2048 embd) needs 0.01-0.05 or output goes garbled.

* The eval callback registration had a bug where it only got set inside the graph-reuse branch: so capture silently stopped working after the first inference. Took a while to track that one down.

* You need \\~500K tokens of activation data for a good SAE. Harry's DPO conversations are \\~14K tokens each, so 20 rows gets you there.

* Persona DPO overfits by step 200 with small datasets. Step 200 was the sweet spot (\\~97% eval accuracy).

* SAEs are not the be-all, end-all of this process and in fact are one of only several pathways to feature interpretability, but they are a simple approach and the process should be fairly adaptable.

Enjoy!