Every picture in this book has been drawn in two dimensions, occasionally three. We have leaned on that relentlessly, and it has served us well: arrows, shadows, parallelograms, the circle becoming an ellipse. And at every turn we reassured ourselves that the arithmetic carries into higher dimensions untouched, that the formulas do not care how many numbers a vector has. That reassurance was true, and it still is. The formulas carry perfectly.
But we quietly promised something else along the way, or at least let the reader assume it: that the intuition carries too. That three hundred dimensional space is just like the plane, only roomier, and that the pictures in our heads keep meaning what they meant. This is where we admit that this second promise does not hold. The formulas travel into high dimensions unharmed. The intuitions do not. They break, sometimes gently and sometimes violently, and a mind trained only on two and three dimensions will confidently expect things that are simply false up there.
This closing chapter is about that gap, the difference between arithmetic that keeps working and intuition that stops being reliable. It is a strange chapter, less about building a new tool than about dismantling false expectations before they cost us, because nearly all real machine learning lives in the high dimensions where our instincts mislead us. Knowing exactly how our low-dimensional intuition fails is not a curiosity. It is protection against a specific set of mistakes that the pictures we have been drawing quietly set us up to make.
The picture
The trouble is that we cannot picture high dimensions at all, and so we do the only thing we can: we imagine the plane, or a room, and silently assume the pattern continues. Most of the time we do not even notice we are doing it. When someone says a data point lives in three hundred dimensional space, we picture a dot somewhere in a room and let that stand in. The stand-in is where the errors come from, because the room is lying to us about what three hundred dimensions are like.
Consider one concrete way the pattern breaks, chosen because it is easy to state and genuinely surprising. In the plane, if you scatter points at random in a square, plenty of them land near the middle. In three dimensions, the same is true of a cube. But as the number of dimensions climbs, something odd happens: the points drift away from the centre and pile up near the edges and corners, until in high dimensions almost every random point is out near the boundary and almost none is in the middle. The centre, which felt so central in the plane, becomes nearly empty. The room in our head, where the middle is a perfectly ordinary place to be, has misled us completely.
Here is another, stranger still. Take two random points in the plane and measure the distance between them; sometimes they are close, sometimes far, with plenty of variety. Do the same in very high dimensions and the distances stop varying. Almost every pair of random points ends up almost exactly the same distance apart, as if the notion of “close” and “far” had quietly collapsed into a single “medium” that applies to nearly everyone. In the plane, some points are near neighbours and some are distant; in three hundred dimensions, everyone is roughly equidistant from everyone else, and the idea of a nearest neighbour, so natural on a page, becomes almost meaningless.
These are not tricks or pathological setups. They are how high-dimensional space genuinely behaves, and they contradict the room in our heads directly. We cannot fix this by picturing harder, because the picturing is exactly the problem; our visual imagination runs on three dimensions and there is no fourth to add. What we can do is learn the specific ways the intuition fails, hold them as explicit warnings, and trust the arithmetic over the picture whenever the two disagree. Up there, the arithmetic is the only reliable guide we have.
The math, built up
This chapter has less arithmetic to build than the others, because its lessons are about phenomena rather than formulas. But one calculation shows precisely where the intuition cracks, and it uses only the length we defined in the very first chapter.
Think about the corners of a cube compared to its faces, measured from the centre. In two dimensions, take a square whose sides sit one unit from the centre. The middle of an edge is one unit away, straightforward. A corner, though, is further, by Pythagoras: \(\sqrt{1^2 + 1^2} = \sqrt{2} \approx 1.41\). The corner is about forty percent further from the centre than the edge. Not dramatic, but already the corners reach out further than the flat sides, which is the diamond-and-circle asymmetry from the norms chapter seen once more.
Now let the dimensions climb, keeping the faces one unit from the centre. In three dimensions the corner distance is \(\sqrt{1^2 + 1^2 + 1^2} = \sqrt{3} \approx 1.73\). In ten dimensions it is \(\sqrt{10} \approx 3.16\). In three hundred dimensions the corners sit \(\sqrt{300} \approx 17.3\) units from the centre, while the faces remain stubbornly at one. The corners have raced away to seventeen times the distance of the faces, and there are vastly more corners than faces to begin with. The cube, which in our imagination is a compact and roughly round sort of thing, is in high dimensions overwhelmingly spikes: almost all of it is out in the corners, absurdly far from the centre, with the familiar middle a tiny and nearly empty core.
That single calculation, corner distance growing as the square root of the dimension while face distance stays fixed, is the arithmetic behind the whole phenomenon. It is not exotic mathematics; it is the length formula from chapter one, applied honestly in three hundred dimensions and refusing to match the picture. The formula never lied. It kept computing lengths exactly as it always had. It was our mental image of a cube, borrowed from three dimensions, that was never a fair likeness of the high-dimensional thing.
Measure it yourself
Rather than build a tool, let us watch the intuition break with our own code, by measuring high-dimensional space and comparing what we find to what the room in our heads expects.
First, the claim that random points drift to the edges. We scatter points at random inside a cube in growing numbers of dimensions and measure their distance from the centre. If the room in our head were right, plenty should sit near the middle, near distance zero:
import numpy as nprng = np.random.default_rng(0)for dim in [2, 10, 100, 1000]: points = rng.random((5000, dim)) -0.5# centred cube, sides at ±0.5 distances = np.linalg.norm(points, axis=1)print(f"{dim:4d} dimensions: average distance from centre = {distances.mean():.2f}")
2 dimensions: average distance from centre = 0.38
10 dimensions: average distance from centre = 0.90
100 dimensions: average distance from centre = 2.88
1000 dimensions: average distance from centre = 9.13
In two dimensions the average distance from the centre is small; the middle is a normal place to be. As the dimensions climb, the average distance grows steadily, and the fraction of points anywhere near the centre falls toward nothing. The points have drifted outward exactly as promised, and the higher we go the emptier the middle becomes.
Now the stranger claim, that distances between points stop varying. We take many random points in high dimensions, measure all the distances between pairs, and look at how much those distances differ from one another:
for dim in [2, 10, 100, 1000]: points = rng.random((500, dim))# distances from the first point to all others d = np.linalg.norm(points[1:] - points[0], axis=1) spread = d.std() / d.mean() # variation relative to the averageprint(f"{dim:4d} dimensions: relative spread of distances = {spread:.3f}")
2 dimensions: relative spread of distances = 0.470
10 dimensions: relative spread of distances = 0.182
100 dimensions: relative spread of distances = 0.057
1000 dimensions: relative spread of distances = 0.017
The relative spread shrinks as dimensions grow. In two dimensions distances vary a good deal, some points genuinely near and others far. By a thousand dimensions the spread has collapsed: nearly every point is almost exactly the average distance from the first, and the difference between the nearest and the farthest has all but vanished. The concept of a nearest neighbour, so solid on a page, has quietly dissolved. We did not derive this; we measured it, and the measurement contradicts the room in our heads as flatly as a measurement can.
Where it lives in ML
The curse of dimensionality is not an abstraction in machine learning; it is a daily constraint that shapes what does and does not work, precisely because machine learning lives almost entirely in the high dimensions where intuition fails. Data routinely has hundreds or thousands of features, which is to say hundreds or thousands of dimensions, and every strangeness of this chapter is in force.
The collapse of distance is the sharpest example. A whole family of methods works by finding the nearest points to a given one, on the reasonable-sounding idea that nearby points are similar. This is the logic of recommending what similar users liked, of classifying a thing by what its neighbours are. But we have just seen that in high dimensions nearly every point is almost the same distance from every other, so “nearest” stops being meaningful, and methods built on it degrade badly as the dimensions climb. The comfortable intuition that near means similar, so trustworthy on a page, quietly stops holding, and a practitioner who does not know this will build a method that works in their small tests and fails on real, high-dimensional data for reasons they cannot see.
The emptiness of high-dimensional space causes a second, related problem: data becomes desperately sparse. To cover the plane with sample points you need only so many; to cover a cube, more; to cover a three hundred dimensional space at the same density you would need more points than there are atoms in anything. Real datasets, however large they feel, are unimaginably sparse in the spaces they inhabit, a scattering of points in a vast emptiness. This is why models need so much data and still see only a thin sample of the space they are meant to handle, and why generalising beyond the sampled points, filling the enormous gaps, is so much of the difficulty of the field.
And yet high dimensions are not only a curse, which is the hopeful turn this book ends the part on. The reason machine learning works at all, despite the emptiness, is that real data almost never fills the high-dimensional space it nominally lives in. It clusters on a much lower-dimensional surface within that space, a sheet or ribbon folded through the high dimensions, occupying far fewer true dimensions than the feature count suggests. This is the low-rank hope made physical, the same structure the singular value decomposition finds and exploits. The curse says the ambient space is impossibly vast and empty; the saving grace says the data does not actually spread through it but huddles on something small enough to learn. Nearly every successful method is, in some form, an attempt to find and work on that hidden low-dimensional structure rather than the hostile high-dimensional space it is embedded in.
Common misunderstandings
The formulas do not break in high dimensions; the intuitions do. The single most important thing to carry out of this chapter is that nothing went wrong with the mathematics. The length formula, the dot product, the norms, every tool still computes exactly what it always computed, correctly, in any number of dimensions. What fails is the mental picture we attach to the numbers, the room in our heads that we borrowed from three dimensions. When the picture and the arithmetic disagree in high dimensions, the arithmetic is right. Distrust the picture, never the formula.
High dimensions are not just “more room.” The natural assumption is that adding dimensions simply gives space more room, the way a cube has more room than a square, and that everything otherwise behaves the same. It does not. Adding dimensions changes the qualitative behaviour of space: where the mass sits, how distances distribute, how full a region can be. It is not the plane with extra elbow room; it is a genuinely different regime where the familiar relationships no longer hold. Treating high-dimensional space as a roomier version of the plane is exactly the mistake this chapter exists to prevent.
The curse does not mean high-dimensional machine learning is hopeless. Reading only the warnings, one might conclude that learning in high dimensions cannot work, yet it plainly does. The resolution is that real data does not occupy the full high-dimensional space; it lies on a much lower-dimensional structure within it, where the curse is far weaker. The methods that succeed are the ones that find and exploit that structure. The curse describes the hostile space data is embedded in, not the friendlier shape the data actually takes. Hopelessness would follow only if data truly filled the space, and it essentially never does.
Dimensionality reduction is not merely throwing information away. Because reducing dimensions sounds like loss, it is tempting to see it as a regrettable compromise. Seen through this chapter, it is closer to the opposite: it is moving from the hostile, empty, intuition-breaking high-dimensional space down onto the smaller true structure where the data actually lives and where distance and nearness mean something again. Done well, it discards the empty dimensions the data was not using and keeps the ones it was, which is why it so often makes models work better rather than worse. It is less a sacrifice than a return to a space our tools and intuitions can handle.
Check your intuition
Try each before opening the answers.
1. In high dimensions, are randomly scattered points more likely to be found near the centre of a region or near its edges? Why does this contradict our intuition from the plane?
2. A method classifies a new data point by looking at its nearest neighbours. Why might this method work well in low dimensions but poorly in very high ones?
3. In a cube whose faces sit one unit from the centre, how far is a corner from the centre in one hundred dimensions? Use the length formula.
4. If high-dimensional space is so hostile and empty, why does machine learning on high-dimensional data work at all?
5. In high dimensions the arithmetic of the length formula and the picture in your head disagree about how a cube behaves. Which do you trust, and what is the general lesson?
TipAnswers
1. Near the edges, overwhelmingly. As dimensions grow, random points drift away from the centre and pile up near the boundary, until in high dimensions almost none are anywhere near the middle. This contradicts the plane, where scattering points in a square leaves plenty near the centre, a perfectly ordinary place to be. Our intuition, trained on two and three dimensions, expects a populated middle, and high dimensions empty it out, because there is vastly more room near the boundary than near the centre once the dimensions climb.
2. In low dimensions, nearest neighbours are genuinely meaningful: some points are close and some are far, so “nearest” picks out truly similar points. In very high dimensions, distances between points collapse toward a single common value, so nearly every point is about the same distance from the new one, and the nearest neighbour is barely nearer than the farthest. The method’s core assumption, that nearby means similar, quietly stops holding, and its accuracy degrades as the dimensions rise even though the procedure itself is unchanged.
3. The corner is \(\sqrt{100} = 10\) units from the centre, while the faces remain one unit away. Each of the hundred coordinates contributes one to the sum under the square root, giving \(\sqrt{1 + 1 + \cdots + 1} = \sqrt{100} = 10\). The corner is ten times further from the centre than the face, a cube that in our imagination is compact but in a hundred dimensions is almost entirely distant spikes with a tiny empty core.
4. Because real data does not fill the high-dimensional space it lives in. It clusters on a much lower-dimensional structure within that space, a folded surface occupying far fewer true dimensions than the feature count suggests, where the curse is far weaker and distances mean something again. The methods that work are the ones that find and operate on that hidden low-dimensional structure. The space is hostile; the shape the data actually takes within it is not, and that gap is what makes learning possible.
5. Trust the arithmetic, always. The length formula computes correctly in any number of dimensions; it is the mental picture, borrowed from three dimensions, that fails to match. The general lesson, and a fitting one to end the part on, is that our geometric intuition is a tool with a limited range, superb in the low dimensions it was built for and unreliable beyond them, while the arithmetic we derived from that intuition outlives it and keeps working where the pictures cannot follow. When they disagree, the formula is the one that has earned our trust.