Oversmoothing Is a Training Dynamics Failure, Not an Architectural One

Deep graph networks stop working past five to eight layers. Node embeddings drift toward one another until nothing is distinguishable, and the standard explanation is architectural: message passing multiplies by a propagation operator over and over, that operator is a low-pass filter, so depth smooths. The explanation is incomplete. Sparse networks whose weights are never trained use exactly the same operators and stay stable out to 20 layers. Something other than the operator is doing the damage.

I started by reproducing Untrained GNN Tickets across three architectures and three citation datasets, sweeping depth from 2 to 20 layers. In that setup the weights stay random and frozen forever, and the only thing learned is a binary mask over connections. It held up. Dense trained models fell apart with depth while the untrained sparse ones did not, and on a 32-layer network the Mean Average Distance diagnostic showed dense embeddings collapsing to near-zero pairwise distance within a few layers while the frozen version kept its spread. The reproduction was not a formality. It gave me a baseline I trusted enough to argue against later.

Then the confound. Untrained tickets differ from dense models in two ways at once: they are sparse, and they never train. Most descriptions of the result credit the sparsity, but with only those two configurations the two properties move together and neither can be blamed. The missing case is a model that is sparse and still trained, which is what Unified Graph Sparsification gives you, learning masks over weights and edges with the gradients still flowing. It collapses with depth much like the dense model does. That one comparison settles the question, and sparsity is not what saves the untrained networks.

So is training itself fatal to depth, or only training left unconstrained? Weight reparameterization suggests the second. Building the effective weights from the covariance of the propagated features, whitening them in effect, keeps representations diverse while training proceeds normally, and it survives depths where dense training died long ago. I ran it against a version that applies the same transform once at initialization, which helped very little. The two differ only in whether the correction keeps being applied, so the failure lives in the trajectory rather than the starting point.

The result I did not expect was that the two fixes do not stack. Applying the structured initialization to the untrained tickets made them much worse. Mask learning appears to need a high-entropy random basis to select from, and structuring that pool leaves less to find. The two methods avoid collapse by different and partly incompatible routes: one refuses to train the weights at all, the other reshapes how they move. I could have left the experiment out for a cleaner story, but it is the part that says something about mechanism.

One smaller finding came from reading code rather than papers. Edge-Popup ranks connections by the absolute value of their learned scores. I tried the raw signed scores instead and got better results at lower sparsity, which hints that the sign carries expressivity the absolute value throws away.

This was a course-scale study on citation graphs, with most conclusions read off accuracy-versus-depth curves and a single smoothness diagnostic, not a theorem about representation rank. What I took from it is narrower and more portable. When two properties are entangled in every published configuration, go run the one nobody ran. And watch the geometry of the representation rather than only the accuracy, since collapse shows up in embedding distances well before the score notices. Depth-induced collapse is not a graph-specific disease, and asking what a network's capacity does to its representations led fairly directly to the work I did afterward on deliberately weakened language models.