Sparse Autoencoders (SAEs) are a type of neural network that efficiently learns data representations by enforcing sparsity, capturing only the most essential data characteristics. This process reduces dimensionality and improves generalization to unseen information.
Language model (LM) activations can be approximated using SAEs. They do this by sparsely decomposing the activations into linear components using…
