Sparse Autoencoders on a Genome Model, and Why I Shuffle Modalities Now

Two things from this project stuck with me, and neither is the number we put on the leaderboard. The first: a sparse autoencoder trained on a DNA language model, with no binding labels anywhere in its objective, produced features that pick out transcription factor binding on held-out genes, at a median best-feature AUROC of 0.659 across 164 factors. The second: when we fed our cross-modal model the wrong protein, AUROC barely moved while average precision fell by half.

Transcription factors switch genes on and off by binding particular stretches of promoter DNA. The maps of where they bind come from ChIP experiments, which are expensive, condition-specific and missing for most factors in most species, so a method that needs those labels only solves the easy half of the problem. BindingBench asks for methods that work without them, which turns the task into a question about representation. How much regulatory structure does a genome model already carry, and can you read it out?

We borrowed the answer from language model interpretability. A TopK sparse autoencoder on SpeciesLM embeddings, 768 dimensions expanded to 6144 latents with 32 active at any position, trained on reconstruction alone. Then for each factor we chose its single best latent on a training split and scored that latent on held-out genes. High-occupancy factors like abf1 and hap5 came out strongest. I would describe the result carefully, though. Picking the best feature per factor after the fact means you need to know which factor you are looking for, so what we showed is that separable features exist inside the representation, not that the model contains a binding-site detector you could deploy. The gap between the information being present and the model having a usable module is something I have become fussy about since.

The conditioning is the part I would most like to build on. Instead of gluing a protein vector onto a DNA vector, we used FiLM: the protein embedding produces scale and shift parameters that modulate a frozen DNA encoder, so the factor's identity changes how the sequence gets read rather than sitting next to it. Against late fusion it won on AUROC, 0.858 to 0.835, and tied on average precision, 0.0107 to 0.0106. From that table alone you would shrug and conclude the fusion strategy hardly matters.

Then we shuffled the proteins, so every promoter was conditioned on some other factor's identity. AUROC went from 0.858 to 0.832, near enough to noise. Average precision went from 0.0107 to 0.0059. The protein pathway was carrying real, factor-specific information and the headline metric could barely see it. Binding sites are vanishingly rare among candidate positions, and with imbalance that severe, AUROC is dominated by the enormous pool of easy negatives while average precision lives where the model has to commit. I now run the shuffle by default on anything with a second input. Whether a model genuinely uses a modality or is coasting on the dominant one is not a question an aggregate score will answer.

None of which makes any of these models useful yet. Average precision hovered around 0.01 across the board, and the classical baselines are not a courtesy: STREME and JASPAR with FIMO come out of decades of work on sequence motifs and stay interpretable in a way a dilated convolutional stack never will. In earlier projects my baseline was a smaller version of my own model. Here it came from a different field entirely and beating it was not a given, which is what got me interested in hybrids that keep the motif prior instead of replacing it.

Leakage in biology also has an unfamiliar shape, following identity and homology rather than rows in a table. We held out whole genes so that a feature could not be scored on the sequence it was chosen from, and whole factors to test generalization to ones never seen. We also had to admit the comparison was not perfectly matched, since runtime limits meant the autoencoder exports used lower peak caps than the supervised models.

What pulled me toward biology is that the failures here are hypotheses. Our best guess for why protein conditioning underperforms is that mean-pooling an ESM-DBP embedding smears out the DNA-binding domain, which is the part that actually determines specificity, and that anisotropy in the embedding space blurs whatever survives. That is a representation problem with obvious next moves: pull out the domain-specific subsequence, or drop the pooling and let the DNA encoder attend to the protein directly. It is also the question I find most interesting in multimodal models generally, which is how to put two separately trained encoders into one space without the conditioning signal collapsing into a generic prior. Doing that where the ground truth is a mechanism rather than a preference label is the part I would like to keep.