import numpy as np
v1 = np.array([1.0, 2.0])
v2 = np.array([3.0, 6.0]) # exactly 3 * v17 Independence, Basis, and Rank
The itch
We have been relying on a concept we never formally defined. Back when we introduced matrices, we stated that every vector is built from the two basis vectors \([1, 0]\) and \([0, 1]\). We also said a transformation is completely determined by where those two specific vectors land. We simply called them the basis and kept moving. It was a mathematical loan, and this chapter pays it back.
The underlying question is one of redundancy, and it is helpful to feel the problem before we define it precisely. Suppose someone hands you a specific set of vectors and claims they are the ultimate building blocks for everything you need. A natural concern should arise. Are all of them actually necessary? Could you throw one away without losing any reach because it was already buildable from the others? Or worse, could the set be missing something important, leaving a region completely out of reach?
Think back to the concepts from the span chapter. We saw that two vectors pointing in different directions successfully span the entire flat plane. However, two vectors pointing the exact same way only span a single narrow line because the second vector added no new direction. That second vector was completely redundant. A collection of vectors can easily contain passengers, which are members that ride along without providing any new mathematical reach. That same collection can also be too small to cover the space we actually care about.
This chapter is about getting that pile of vectors exactly right. We want to know when a set has absolutely no passengers. We want to know when it is the perfect size to build a space with nothing wasted and nothing missing. Finally, we want to know how to measure the true reach a set actually has. These three questions correspond to the terms independence, basis, and rank. They are really just one central question about redundancy asked in three different ways.
The picture
We can start by making the worry visual. Imagine two vectors that point in distinctly different directions. Between them, they can reach the entire plane. Every possible point is just some combination of those two arrows. Neither vector is redundant because neither can be built from the other. If you take either one away, your reach collapses from the whole plane down to a single line. Both vectors are pulling their weight.
Now imagine two vectors that point the exact same way, where one is just a stretched copy of the other. Together, they only reach the single line they both rest on. The second vector brings absolutely nothing new because it is already a simple scaling of the first. It is a passenger. A set containing a passenger is called dependent. At least one member depends on the others because it can be built from them and adds no new reach. A set with no passengers, where every single member contributes a unique direction, is called independent.
There is a clean visual way to see this difference that will matter when we look at the math. Take a dependent set, like the two vectors sharing the same line. Because the second is just a scaled copy of the first, we can combine them and arrive perfectly back at the origin. We just walk out along one arrow and walk backward along the other until they cancel each other out. For a truly independent set, this is physically impossible. The only way to combine two vectors pointing in genuinely different directions and land back at the origin is to use absolutely nothing of either vector. So, independence holds a very sharp test. Can a combination reach zero using real amounts, or can only the trivial all-zero combination achieve it?
Once we understand independence, the concept of a basis is the natural next step. A basis is simply an independent set that also manages to reach everything. It provides enough vectors to build the entire space without a single extra one wasted. In the flat plane, any two independent vectors form a valid basis because two independent vectors already span the whole plane. The familiar \([1, 0]\) and \([0, 1]\) are just one basis. They are the simplest one, but certainly not the only one. Any two arrows pointing in different ways will work perfectly. A basis is a complete set of building blocks with absolutely zero waste. It is independent, so there are no passengers, and it is spanning, so nothing remains out of reach.
Rank is simply the counting version of this exact idea. Given a set of vectors, or the matrix holding those vectors as columns, the rank measures how many independent directions they actually reach. It counts the number of building blocks that genuinely pull their weight once all the redundant passengers are removed. Three vectors that all lie perfectly flat in a plane have a rank of two, not three. Despite their headcount, they only reach a two-dimensional plane, proving one of them is a passenger. Rank ignores the raw number of vectors and measures the true mathematical reach.
The math, built up
The visual picture gives us a sharp definition of independence, so we can now translate it into math. A set of vectors \(\mathbf{v}_1, \mathbf{v}_2, \ldots, \mathbf{v}_k\) is independent when the only linear combination that equals the zero vector is the one where every scaling amount is strictly zero:
\[ c_1 \mathbf{v}_1 + c_2 \mathbf{v}_2 + \cdots + c_k \mathbf{v}_k = \mathbf{0} \quad\text{forces}\quad c_1 = c_2 = \cdots = c_k = 0. \]
You can read this directly through our earlier picture. Reaching the zero vector means walking out along the arrows and arriving perfectly back at the origin. If the only way to do that is to use exactly none of them, then no vector was secretly canceling against the others. This proves none of them are passengers, confirming the set is independent. If you can use actual non-zero amounts and still land at zero, then one vector was completely expressible using the rest, proving the set is dependent. The entire concept of redundancy is captured perfectly by this one equation checking for zero.
A basis for a space is an independent set that successfully spans it. These two strict conditions naturally pull against each other. The spanning rule demands enough vectors to reach everything. The independence rule demands few enough vectors that none are wasted. A basis sits exactly where those two rules meet perfectly. This creates a mathematical fact that feels surprising at first but becomes obvious. Every single basis for a specific space has the exact same number of vectors. You cannot build the flat plane with three independent vectors because there is simply no room for the third one; it will always depend on the first two. You also cannot span the plane with just one. Therefore, every basis for the plane contains exactly two vectors. That fixed number is the dimension of the space. We finally have a precise definition for a word we have used loosely. Dimension is simply the size of any valid basis.
Rank applies this exact same counting logic to whatever specific set of vectors you happen to have, even if they do not form a basis. The rank of a set of vectors is the exact size of its largest independent subset. It counts the genuine independent directions hidden among them. If the rank perfectly equals the total number of vectors, the set has no passengers and is fully independent. If the rank is lower than the number of vectors, some are completely redundant. For a matrix, the rank measures how many independent directions its columns can reach. This reveals the true dimensionality of the space the transformation can output, completely ignoring how many columns are actually written down.
Build it yourself
Independence, rank, and dimension all rely on counting genuinely independent directions, and NumPy will handle this counting for us. First, we will catch a dependent set in the act to see the definition working.
Here are two vectors sitting on the exact same line. The second is a perfect multiple of the first:
Our definition stated that a set is dependent when some combination reaches the zero vector without using all zeros. We can easily see the required amounts here. Three of \(v_1\) minus one of \(v_2\) cancels out perfectly.
print(3 * v1 - 1 * v2)[0. 0.]The output is perfectly zero, and the scaling amounts were \(3\) and \(-1\). This non-trivial path back to the origin proves dependence perfectly. For a truly independent pair, no such amounts exist. Only \(0\) and \(0\) would land on the zero vector.
Counting the independent directions inside a set is exactly what rank measures. NumPy computes this directly:
dependent = np.column_stack([v1, v2])
print(np.linalg.matrix_rank(dependent))1The rank is exactly one. We wrote down two distinct vectors, but they only provide one independent direction because the second is a clear passenger. Now we will test an independent pair:
w1 = np.array([1.0, 2.0])
w2 = np.array([2.0, 1.0]) # not a multiple of w1
independent = np.column_stack([w1, w2])
print(np.linalg.matrix_rank(independent))2The rank is exactly two. We have two vectors and two genuine directions with no passengers. These two perfectly form a basis for the plane. When the rank matches the total number of vectors, the set is formally independent.
This tool easily sees through a more deceptive case. Here are three vectors in a three-dimensional space. At first glance, they look highly independent because the lists look very different:
a = np.array([1.0, 0.0, 0.0])
b = np.array([0.0, 1.0, 0.0])
c = np.array([1.0, 1.0, 0.0]) # a + b, a hidden passenger
M = np.column_stack([a, b, c])
print(np.linalg.matrix_rank(M))2The rank is exactly two, not three. The third vector is simply \(a + b\). It is entirely buildable from the first two, so despite looking different, it adds zero new direction. All three vectors are trapped inside the same flat plane within the larger space. The rank reports the mathematical truth our eyes missed. There are only two independent directions. This perfectly highlights why rank is so important. In massive datasets, we cannot visually spot the passengers. A set can look massive while secretly collapsing into a tiny space. Rank honestly counts what is truly there.
Where it lives in ML
Redundancy among features is a very common and quiet problem in real-world data, and rank is how we detect it. Imagine a dataset has one column for temperature in Celsius and another column for the exact same temperature in Fahrenheit. You have two columns, but one is just a strict transformation of the other. It is a passenger in the exact mathematical sense we just defined. It adds absolutely zero independent information. A model receiving both is carrying a totally redundant direction. The matrix of features is rank-deficient because it has fewer independent directions than physical columns.
This is far more than a mathematical curiosity; it actively breaks systems. Several standard machine learning methods firmly assume the feature directions are independent. When they are not, the algorithms become unstable or fail completely. A clean example is fitting a linear model. Under the surface, this involves mathematically undoing a transformation. That undoing is only physically possible when the relevant vectors are truly independent. If you feed the system redundant features, the undoing process has no unique answer. A dependent set simply cannot pin down one single solution. The model cannot decide how to split its influence between two columns saying the exact same thing. The result is wildly arbitrary weights. Detecting redundancy and verifying the rank is a mandatory part of preparing data honestly.
Rank also names a highly desirable goal. A massive dataset might have a thousand recorded features but a significantly lower true rank. The data looks incredibly high-dimensional on the screen, but it actually lives in a much smaller number of independent directions, with the remaining data built from those core features. This is the exact hope behind dimensionality reduction. If the true mathematical rank is low, we can discard the redundant directions and keep only the ones carrying real information. We can shrink a thousand numbers down to a few dozen with almost zero loss in quality. This entire concept relies on the gap between how many features are written down and how many independent directions actually exist. When we reach the topics of PCA and SVD later in the book, this gap is the exact property we exploit.
There is a deeper appearance of rank in the study of neural networks. The transformations a network learns naturally have a specific rank. A low-rank transformation actively collapses its input into a much smaller space, permanently discarding the rest of the data. Determining whether that specific collapse is helping the model learn or quietly destroying vital information is an active question in the field. Understanding that behavior requires the exact notion of rank we just built. We must know how many independent directions survive a transformation and how many are lost.
Common misunderstandings
Independence describes the entire set, not just pairs. It is very tempting to check for independence by looking at the vectors two at a time. If no vector is an obvious multiple of another, you might assume the whole set is safe. This fails completely. Three vectors can easily be non-multiples of each other while still remaining dependent. The vector \([1, 1]\) is not a multiple of \([1, 0]\) or \([0, 1]\), yet it is exactly their sum. Therefore, the three together form a dependent set. Independence asks if any combination of all the vectors reaches zero. You cannot verify this just by scanning pairs.
A basis is never unique. The standard basis of \([1, 0]\) and \([0, 1]\) is so incredibly common that it feels like the only true coordinates of the plane. This is incorrect. It is just one basis among an infinite number of choices. Any two independent vectors form a valid basis, providing a completely different set of coordinates for the exact same physical space. The standard basis is simply convenient, not mathematically privileged. A massive amount of power in linear algebra comes from deliberately choosing a strange basis that makes a complex problem simple. That trick only makes sense once you realize the standard basis holds no special power.
Rank counts independent directions, not vectors or numbers. Rank is incredibly easy to confuse with simpler counting tasks. It does not count the total number of vectors, which easily exceeds the rank when passengers exist. It does not count the number of non-zero entries or the columns that happen to look different visually. Rank strictly counts the number of genuinely independent directions. This requires mathematically seeing through combinations rather than just scanning the raw data. This is exactly why we need a computer function to measure it in high dimensions where human eyes fail.
Dimension applies to the space, while rank applies to your specific set. These terms blur together because they match perfectly in the ideal scenario. An independent set spanning the plane has a rank of two, and the plane has a dimension of two. However, they answer fundamentally different questions. Dimension asks how many independent directions the space has room for. Rank asks how many independent directions your specific pile of vectors actually reaches. A set of vectors floating in three-dimensional space can have a rank of one, two, or three. The dimension of the space is permanently locked at three. The rank of your set simply depends on what vectors you brought with you.
Check your intuition
Try to answer these questions before opening the answers.
1. Are the vectors \([2, 0]\) and \([0, 5]\) independent? Are \([2, 0]\) and \([4, 0]\)? Answer from the picture, not a computation.
2. A set contains the zero vector along with two other, genuinely different vectors. Can the set be independent? Think about what combination could reach the origin.
3. Three vectors live in the plane, ordinary two-dimensional space. Can they be independent? What does this say about their rank?
4. A matrix has four columns, but its rank is two. In plain terms, what does that tell you about the columns?
5. You measure the rank of a \(1000 \times 1000\) matrix of data and find it is \(40\). What does this suggest about the data, and why might you be pleased to discover it?
TipAnswers
1. The first pair is completely independent. The vector \([2, 0]\) points perfectly along the horizontal axis, and \([0, 5]\) points perfectly along the vertical axis. They point in two different directions, and neither is buildable from the other. The second pair is fully dependent. The vector \([4, 0]\) is just a scaled copy of \([2, 0]\). Both sit on the exact same line, making the second one a passenger.
2. No, the set can never be independent. The mathematical zero vector is always a passenger. You can take any physical amount of the zero vector, take absolutely none of the other vectors, and still sit perfectly at the origin. Five copies of the zero vector still equals zero. That provides a non-trivial combination reaching zero, which is the exact definition of dependence. Any set containing the zero vector is automatically dependent.
3. No, three vectors in a flat plane can never be independent. The plane has a permanent dimension of exactly two. This means any valid basis only holds two vectors, and no independent set can ever be larger than a basis. A third vector placed in the plane is always buildable from the first two. Therefore, three vectors in a plane have a maximum rank of two. At least one must be a passenger.
4. Two of those four columns are completely redundant. The four columns only manage to reach two independent directions. Whatever the other two columns happen to be, they are entirely built from the first two and add zero new mathematical reach. Despite having four columns of data to work with, the matrix can only produce a two-dimensional output space.
5. The massive data file has a thousand recorded features but only forty genuinely independent directions. The other nine hundred and sixty columns are just hidden combinations of those core forty. The data looks incredibly high-dimensional on the screen but truly lives in a very small mathematical subspace. You would be thrilled because this data is highly compressible. You can keep the forty directions and completely discard the rest with almost zero loss. A low rank inside a large matrix is a massive opportunity for optimization.