What Sparseness Means for Brains, AI, and Learning

By Daniel J. Graham

From early papers that helped spark the deep learning revolution, to defining modern physiological studies of the visual system, to the landmark paper that introduced DeepSeek to the world in 2025, the terms sparse and sparseness appear a lot in neuroscience and AI.

Here I will lay out what sparseness is and where it comes from. I will argue that it is both a more subtle and a more powerful idea than is usually appreciated. And I will suggest that sparseness is a signature of learning that is likely to be found in any system—biological or machine—that builds efficient representations of the environment.

What is sparseness?

The textbook definition of sparseness is that it is what is shown by an encoding system with low activity ratios. This property takes two general forms. In a sparse system, only a handful of units should be “on” or highly activated to represent a given input. This is called population sparseness. Over time, all units will participate. But each unit will be used relatively rarely. This is known as lifetime sparseness. Systems described as sparse typically show high levels of both population and lifetime sparseness.

The first point I want to make is that the sparseness of a system is always sparseness for some set of inputs. There is no code that is sparse for any possible input. And the price of sparseness is inefficiency for all but the kinds of inputs you were trained on.

The idea of sparseness was introduced by Horace Barlow in 1972 to describe something similar to population sparseness. But he didn’t make the connection to the nature of the inputs. This had to wait until the work of David Field. His 1987 paper on the basic statistics of natural scenes paper sparked the natural scenes movement in visual neuroscience, and led to our modern understanding of sparseness. In this paper, Field showed that natural scenes were regular in their basic spatial statistics. And he started using the term sparseness. In 1994, Field showed, using numerical simulations, that V1-like representations of natural scenes led to sparse patterns of activation.

With his post-doc Bruno Olshausen, Field would go on to build a generative model that optimized a neural net-type coding system for sparseness. Their 1996 model showed that V1- like representations emerge for free in a representational system whose only goals are to faithfully represent a collection of natural image inputs and to do this with the fewest non-zero basis functions for each patch.

What is remarkable about this result is that it required no supervision. When you force a code to be sparse, efficient representations simply fall out. As a bonus, you get solutions that look quite obviously biological. These two outcomes are not a coincidence. ¹

Evidence that the brain is sparse

There is a wealth of evidence from estimates of metabolic costs that the brain must be sparse. Spiking is just too energetically costly for brains to have more than a small fraction of neurons active at once. And we know that having a large population of neurons active all at once is bad news: it’s a seizure.

There is also strong physiological evidence that the mammalian brain is sparse in response to natural images. In a landmark paper from the Allen Institute, Saskia De Vries and colleagues measured sparseness using optical imaging of visual cortex in behaving mice while they viewed natural scenes and movies. As they report, “across all areas, layers, and stimuli, visual responses in mouse cortex were highly sparse.”

Considering both the estimated bounds on metabolism and the physiological results, it may not be far from the truth to say that we really do use only 10% of our brains—at a time.

The chief criticism of the idea that the brain is sparse is that it doesn’t predict the coding strategies of the earliest stages of vision. If the brain is optimized for sparseness, the argument goes, why don’t we see the characteristic Gabor filter-like properties already in the retina and thalamus? Across vertebrate retinas, we see almost universally that filters have rotationally- symmetric difference-of-Gaussians-type structure. This is a fair point, even though retinal filters are in fact sparser in their responses than other kinds of filters that accomplish the same level of decorrelation of the input.

Another criticism is that maximizing sparseness globally leads to a variety of the dark room problem: the best way to minimize energy use is to be dead.

I think the answer here is that sparseness is a general, overarching goal that brains employ. It is not the only thing brains try to achieve, and it is not necessarily fully optimized. But it is something all brains aim for to a greater or lesser extent.

This is because our environment is sparse, and any system that wants to efficiently understand our environment will need to be sparse too.

Exploiting sparseness

Animals (and machines) live in extremely high dimensional environments. Yet they only need to extract a tiny fraction of what’s out there. The structure relevant to survival is sparsely distributed. Because the world contains spiky data, the best solution is to be sparse yourself. Once a system—biological or machine, supervised or unsupervised—finds this structure, it will display sparseness.

Early supervised convolutional networks like ConvNet and LeNet—which predate Olshausen and Field’s 1996 sparse coding paper—learned their input filters from the inputs themselves. This was an important innovation. And these models tended to learn kernels that resemble the Gabor-like filters that sparse coding generates.

Sparse coding explains why filters learned by early CNNs look like those that emerge in unsupervised sparse coding. It’s because becoming sparse is a signature of efficient learning. In learning almost any task from natural inputs, you will almost inevitably arrive at sparse representations, even without explicitly optimizing for sparseness.

Sparseness is not about finding the main axes of variation in a set of data. That’s what is accomplished by principal components analysis (and, equivalently for statistically stationary inputs, a Fourier code). There is a famous figure in Field’s 1994 paper “What is the goal of sensory coding?” that illustrates this:

Whereas the principal axes capture the data well in (B), they fail to align with the true structure of the sparse data in (A), even though they are arrived at through the same statistical analysis as in (B). Coefficients of basis vectors P1 and P2 are both likely to be moderately active for any input in both (A) and (B). But if the basis vectors are aligned with the actual structure of the data (S1 and S2), data can be well characterized by just one value: where on S1 or S2 it lies. So any given input is likely to have one large coefficient, and one that is effectively zero. This can be extended to many dimensions. ²

Ralph Waldo Emerson said the ends preexist in the means. One could also say that sparseness preexists in understanding the environment.

The sparse environment

Matching the sparseness of the environment has downsides. One is being unable to deal with very unnatural unsparse inputs.

There are far more possible images than there are particles in the universe. Only a small fraction of these possible images resemble natural images in their statistics. Plenty of coding schemes, such as a Fourier code, could capture this entire set without a problem.

But this is not how brains—or indeed CNNs—work. Instead, they use sparse codes. Each part of an image is encoded by a relative handful of units. In the brain, these are V1 neurons. In neural networks, they are convolutional units. Neither resembles a dense Fourier code. This is because the brain and CNNs use codes that are only good for the images that they actually encounter, which are natural images. This is why they fail at tasks that involve very unnatural images. I’ll give you an example.

If our brains are matched to our environment, images that deviate strongly from the natural world in terms of their basic statistics should be difficult to see. Look at the two images below. Are they the same or different?

If you study one corner of both images, you can probably see that small patches of the image look different. What you will not notice is that the images are identical, except that the one on the right has been rotated 90 degrees.

These images are statistically random white noise: I chose each pixel value randomly and independently from a Gaussian distribution. The images lack the basic regularities found in natural scenes, specifically sparseness (but also pairwise correlations and scale invariance). Any two randomly-selected white noise images will be statistically very different from each other, and from natural scenes. But they will be indistinguishable to our eyes. They are, in a way, un-visible.

LLMs also fail to spot that the images above are identical (though it depends somewhat on the prompt). And they have trouble making white noise images. When I asked ChatGPT Pro to make me some Gaussian white noise (below left), the images had pixel histograms that were not remotely Gaussian. And the pixel correlations (gauged by their rotationally-averaged spatial frequency amplitude spectrum, inset) deviate pretty strongly from my algorithmically generated white noise (below right), and in fact show something closer to the 1/f structure typical of natural scenes. ChatGPT uses a diffusion model to make an image that has random textures, but this won’t result in a truly random image. You can probably even “feel” that the image on the left is more natural than the white noise.

Deep learning models have traces of the natural world even when they are trying to make something random. That’s how ingrained natural statistics—and sparseness—are in them. ³

Sparseness as a signature of learning

Sparseness shows that a system has learned to represent an input space efficiently.

But becoming sparse takes a considerable amount of energy. As you learn, it is not so easy to assign a given input to different coding units. This is why sparse coding is nonobvious as a general solution to the problem of learning—it starts off being very costly.

Let’s say you are trying to encode images of animals. Initially, an object like a dog might be assigned to some set of sparse units. Over time, other objects are presented, each with their own sparse encoding. Soon the dog could need to be reassigned to a totally different set of units that have no overlap with the initial set, in order for the overall representation to work and be sparse.

This is costly, and it’s very different from learning under a dense code, like Fourier. In this case, learning is just a matter of tiny tweaks in the usage of essentially all coding units as new inputs are encountered, since most or all coding units participate at least a little bit to encode all inputs.

Learning something new is the second-most energetically expensive discretionary task the brain performs (moving your body is first). And we spend many years learning how to operate at a basic level in our environment.

The challenge of learning is primarily in the difficulty of sorting out which handful of units is best at representing a given input before all relevant inputs are encountered. At some level, the enormous cost of training deep learning systems boils down to the same problem.

But once sparseness is achieved, everything becomes nice and efficient. This is a general principle that captures the fundamental nature of learning. It’s hard until it’s easy.

Sparseness and the future of AI

Sparseness is by no means the only idea that makes deep learning work. But the pioneers of CNN systems initially acknowledged the crucial linkages between sparse coding and CNNs. Yet over time the importance of sparseness has been largely forgotten. Its influence has been overshadowed by the power of massive compute.

It’s true that modern transformer architectures are not designed to be sparse. It is certainly possible to build intelligent machines that are not particularly sparse. But we are now seeing the consequences of this design choice. Transformers use world-changing amounts of energy. The gigawatts and gigadollars being spent on compute may prove untenable. These enormous costs are borne not just during training but also during inference and deployment.

The jury is still out on whether architectures like DeepSeek’s mixture-of-experts model—which has sparse utilization as an explicit goal—will take over the marketplace. It’s possible that quantum computing will be commercialized in time to mitigate the exploding compute costs of transformers. Disciples of the bitter lesson may well be proven right again.

But I suspect sparseness will win the day. Mixture-of-experts models have a genuine principle of biological learning behind them—just as the early CNNs did—namely sparseness. Their success is not just about leveraging more and more massive compute. Of course, compute is important. But compute on its own may prove to be self-limiting. I think longer term success depends on actual, deep principles, such as sparseness.

The brain is sparse. Evolution makes it so. We may find that, for intelligent machines to evolve, they too will require sparseness.

This post is adapted from the book-in-progress Sparse by Daniel J. Graham and David Field. Thanks to David Donoho, Bruno Olshausen, Barbara Finlay, Pamela Reinagel, Eero Simoncelli and Joseph Atick for helpful discussions.

Daniel J. Graham is professor of psychological science at Hobart and William Smith Colleges and the author of An Internet in Your Head: A New Paradigm for How the Brain Works (Columbia University Press, 2021). His research focuses on models of message flow on brain networks and on brain systems related to vision coding and human visual aesthetics.

1. 1 I should note here that the basic idea of sparse coding was simultaneously and independently developed by Tony Bell and Terry Sejnowski to solve a different problem, namely blind-source separation. Their approach, called independent components analysis (ICA), is more general than but mathematically equivalent to Olshausen and Field’s sparse coding system. See this book for a primer on the mathematical foundations of ICA and sparse coding.

2. 2 Note also that S1 and S2 are not orthogonal. This is why the “independent” in independent components analysis is something of a misnomer.

3. Of course, if you ask an AI to make a random image from scratch algorithmically, it can do it. The point is that its image model is tailored to our sparse environment.