A merged contribution to SAELens: loading JumpReLU sparse autoencoders trained with dictionary_learning either failed outright or, before v6.23, silently ran them as plain ReLU SAEs. The fix, the evidence, and the tests that pin it.

Context

SAELens is an open-source library for training and analysing sparse autoencoders (SAEs) on language models. Researchers publish trained SAEs in many formats, and SAELens converts them on load. A converter has to get the architecture, the learned weights and any extra parameters exactly right, or the SAE still loads but computes something different from what was trained.

This contribution is about one such converter: the one for SAEs trained with dictionary_learning, including the JumpReLU SAE suites in SAEBench.

Two failure modes, one cause

The converter mapped JumpReluAutoEncoder to architecture="standard". Since v6.23.0 the loader also copies the per-latent threshold into the state dict, so those SAEs could no longer be loaded:

RuntimeError: Error(s) in loading state_dict for StandardSAE:
    Unexpected key(s) in state_dict: "threshold".

Before v6.23 the same SAEs loaded without error but ran as plain ReLU SAEs with the threshold dropped, giving silently wrong latents and L0. The loud failure is the better of the two. The silent one is the dangerous one, because nothing tells the user their feature activations are off.

The fix

Two changes, both needed:

  1. Map JumpReluAutoEncoder to architecture="jumprelu", so the threshold loads into a JumpReLU SAE instead of a standard one.
  2. Set apply_b_dec_to_input=False for it. dictionary_learning computes x @ W_enc + b_enc without subtracting b_dec. Mapping the architecture alone loads cleanly but gives wrong latents whenever b_dec is nonzero.

A load that succeeds is not evidence that the SAE is correct.

What the second change is worth

Measured on a real SAEBench SAE (pythia-160m-deduped, layer-8 residual activations over 152 tokens):

Loader L0 Explained variance
This PR 20.8 0.976
Architecture mapping only 206.8 0.221

One SAE and 152 tokens: a sanity check on this loader, not a benchmark. L0 is the average number of latents active per token; explained variance is the share of the activation's variance the SAE reconstructs.

Tests

Two tests, both failing on main. One checks the converted config is jumprelu with the per-latent threshold kept. The other checks that SAE.from_pretrained reproduces dictionary_learning's encode and decode exactly, with inputs chosen so the threshold actually zeroes some positive pre-activations. Without that choice, a wrong threshold would pass unnoticed.

The change is +109 / −1 across two files. Merged 1 October 2026.