MathIntermediate

Matrix Rank and Linear Independence

Key Points

  • Matrix rank is the number of pivots after Gaussian elimination and equals the dimension of both the column space and the row space.
  • A set of vectors is linearly independent if and only if the matrix formed by placing them as columns (or rows) has full column (or row) rank.
  • The Rank–Nullity Theorem states that rank(A) + nullity(A) = number of columns, which measures degrees of freedom in solutions.
  • A linear system Ax = b is solvable if and only if rank(A) = rank([A|b]); otherwise it is inconsistent.
  • Over a field (reals, rationals, or modulo a prime), Gaussian elimination runs in O() time for an n × n matrix.
  • Competitive programming often needs rank over R, modulo a prime p, or over GF(2) (XOR basis).
  • Numerical stability for real matrices improves with partial pivoting and a careful epsilon to treat near-zero values.
  • The number of solutions to Ax = b (if consistent) is infinite unless rank equals the number of columns; its solution space has dimension equal to nullity.

Prerequisites

  • Vectors and vector spacesRank and independence are defined in the context of spans, subspaces, and bases.
  • Systems of linear equationsUnderstanding Ax = b and solution sets is essential for interpreting rank and nullity.
  • Matrix operations and row reductionGaussian elimination is the standard method to compute rank and RREF.
  • Modular arithmetic and fieldsRank over GF(p) requires modular inverses and field properties (p must be prime).
  • Bitwise operations and XORRank over GF(2) maps naturally to XOR bases for efficient competitive programming solutions.
  • Floating-point arithmetic and numerical stabilityPartial pivoting and epsilon thresholds mitigate round-off errors in real computations.
  • Big-O complexityAnalyzing algorithmic cost (e.g., O(n^3) elimination) helps choose feasible approaches.

Detailed Explanation

Tap terms for definitions

01Overview

Matrix rank measures how many directions a matrix’s columns (or rows) truly span. Intuitively, consider columns as arrows in space: some arrows may be redundant because they point in directions already covered by others. Rank counts the number of indispensable arrows. Computationally, we reveal rank by performing Gaussian elimination: each pivot we find is a new independent direction, and the total number of pivots equals the rank. Importantly, rank is the same whether we look at columns or rows, tying together the concepts of column space and row space. Rank is central to understanding linear independence, solvability of linear systems, and dimensions of solution spaces. If you place k vectors as columns of a matrix, they are linearly independent exactly when the rank is k. For linear systems Ax = b, comparing the rank of A and the augmented matrix [A|b] determines whether the system is solvable; the number of free variables is the nullity (number of columns minus rank), representing degrees of freedom in the solution. In practice, we compute rank differently depending on the setting: using doubles with partial pivoting (real numbers), modular arithmetic when the field is GF(p) for a prime p, or bitwise XOR tricks when the field is GF(2). Across all these cases, rank captures the same geometric idea—the dimension of the span.

02Intuition & Analogies

Imagine you’re building a playlist of songs for a party, but you don’t want duplicates of the same vibe. Each song represents a direction (a musical style). If a new song brings a totally fresh vibe, you keep it; otherwise, you skip it because it’s already covered. The number of genuinely different vibes in your playlist is like the matrix’s rank: it is the count of unique directions that your columns add. Now think of stacking colored transparencies. Each transparency adds some pattern to the final image. If a new transparency is just a blend of the previous ones, it doesn’t add any new pattern. Only those that introduce a new visible pattern increase the overall complexity—these are your pivots. The final count of such impactful transparencies gives you the rank. For linear independence, imagine assembling furniture with a set of tools. If some tools can be mimicked by combinations of others (like using adapters, extensions, and multipurpose tools), then that tool isn’t truly necessary. A set of tools is independent if none can be recreated by the others—in matrix terms, the columns are independent exactly when the column rank equals the number of columns. In solving Ax = b, picture b as a target recipe and columns of A as ingredients. If your pantry (the span of columns) can make that exact taste (b), the system is solvable. If not, no combination of your ingredients can match it. When it is solvable, the number of different ways you can make the recipe corresponds to the number of free variables—the nullity—your degrees of freedom.

03Formal Definition

Let A be a matrix over a field . The column space (A) is the subspace of spanned by the columns of A; the row space (A) is the subspace of spanned by the rows of A. The rank of A, denoted (A), is defined as the dimension of (A) (equivalently, the dimension of (A)). A finite set of vectors \{, , \} is linearly independent if the only solution to is = = = 0. If we form A by placing as columns, then the set is independent if and only if (A) = k. The rank can be computed as the number of pivots in the row-echelon form (REF) or reduced row-echelon form (RREF) of A obtained via Gaussian elimination. The null space (kernel) of A is (A) = \{x : , and its dimension is the nullity, satisfying the Rank–Nullity Theorem: (A) + (A) = n. For linear systems , the Rouché–Capelli theorem states the system is consistent if and only if (A) = ([Ab] is the augmented matrix.

04When to Use

Use matrix rank and linear independence whenever you need to measure how many truly distinct directions or constraints a problem contains.

  • Solving linear systems: Determine whether solutions exist and how many degrees of freedom there are. If rank(A) = rank([A|b]) = r, then there are n - r free variables.
  • Geometry and graphics: Check whether given vectors form a basis for a subspace, compute dimensions of spans, and detect degeneracy (e.g., are points collinear or coplanar?).
  • Competitive programming: Count independent constraints, verify if vectors modulo a prime are independent, compute ranks efficiently over GF(2) using XOR bases, and determine the number of distinct XORs representable by given numbers.
  • Data analysis: Evaluate redundancy among features (columns) or detect whether data lie in a lower-dimensional subspace.
  • Control and networks: Assess controllability/observability (rank tests) and network flow linear dependencies.
  • Algebraic manipulations: Simplify systems, extract bases for row/column spaces, and compute null spaces to describe all solutions. Over reals, use partial pivoting; over a prime modulus, use modular inverses; over GF(2), use bitwise operations for speed.

⚠️Common Mistakes

  • Ignoring pivoting and epsilon: Over reals, failing to use partial pivoting or a proper epsilon leads to dividing by near-zero values and unstable results. Always pick the largest absolute pivot in the current column and treat values with |x| < \varepsilon as zero.
  • Using Gaussian elimination modulo a composite: Modular inverses may not exist when the modulus isn’t prime. Over composite moduli, elimination can fail; either factor the modulus and use CRT or work over a field (prime modulus).
  • Miscounting rank: Counting nonzero rows before fully eliminating (or without handling numerical noise) can overestimate rank. Only count rows with a true pivot (or after RREF with an epsilon test).
  • Mixing rows vs. columns: Linear independence of columns requires comparing rank to the number of columns; independence of rows compares rank to the number of rows. Be explicit about orientation.
  • Assuming nonzero determinant test applies to non-square matrices: det(A) \neq 0 ↔ full rank applies only to square matrices. For rectangular matrices, use rank directly.
  • Overflow and type issues: When working modulo p, ensure all arithmetic is reduced mod p and uses 64-bit where needed. For GF(2) XOR bases, beware of signed shifts and use unsigned types. For floating point, consider scaling or long double when magnitudes vary widely.
  • Forgetting augmented-rank check: To test Ax = b consistency, you must compare rank(A) and rank([A|b]). Checking rank(A) alone is insufficient.

Key Formulas

Rank equals column/row space dimensions

Explanation: The rank equals both the dimension of the column space and the row space. It is the number of independent directions captured by the matrix.

Rank–Nullity Theorem

Explanation: For an m×n matrix A, the rank plus the dimension of the null space equals the number of columns. This tells you how many free variables any linear system has.

Independence via rank

Explanation: Place the vectors as columns and compute the rank. If the rank equals the number of vectors, none of them is a combination of the others.

Pivots equal rank

Explanation: Each pivot reveals a new independent direction uncovered by elimination. The total pivot count equals the rank.

Transpose invariant rank

Explanation: Row and column ranks are equal. Transposing a matrix does not change its rank.

Rank bound

Explanation: A matrix cannot have more independent columns than rows (or vice versa). The maximum possible rank is the smaller dimension.

Submultiplicative rank

Explanation: The product cannot have more independent columns than either factor. Composition cannot increase the number of independent directions.

Full rank and determinant

Explanation: For square matrices, a nonzero determinant is equivalent to full rank and invertibility.

SVD rank characterization

Explanation: The rank equals the number of nonzero singular values from the singular value decomposition. Useful for theoretical insights and numerical analysis.

Rouché–Capelli

Explanation: A system has at least one solution precisely when adding b as a column does not increase the rank.

Dimension of sum of subspaces

Explanation: When combining two subspaces, their dimensions add but you must subtract the overlap to avoid double-counting.

Gram matrix rank

Explanation: Over the reals, the Gram matrix preserves rank. It links inner products to independence.

Complexity Analysis

Gaussian elimination for an m×n matrix performs O(min(m,n)) pivot steps, and each pivot step can touch O(mn) entries when implemented as full elimination (RREF), yielding O(min(m,n)·m·n) time; for square n×n matrices this simplifies to O(). Space usage is O(mn) to store the matrix. If you only eliminate below the pivot (REF), the leading term remains cubic for square matrices. With partial pivoting, the asymptotic cost is unchanged but numerical stability is greatly improved. Over a prime modulus p (GF(p)), modular Gaussian elimination has the same O() time for n×n matrices, dominated by modular multiplications and additions. Computing modular inverses via fast exponentiation costs O(log p) per inverse, but you can reuse a pivot’s inverse across the row, keeping the overall complexity O( + r·log p), where r is the number of pivots (≤ min(m,n)). Space is again O(mn). Over GF(2) with 64-bit integers, maintaining a linear XOR basis of size at most 64 takes O(64^2) per insertion in the worst case, but with bit operations it’s extremely fast in practice. Using bitsets for higher dimensions, elimination on k vectors of dimension d costs roughly O(k· / w), where w is the machine word size (e.g., 64), because bitwise operations process w bits at once. Augmented-system processing for solving has the same cubic time as elimination; extracting a particular solution and a null-space basis costs at most an additional O(mn). Overall, memory is dominated by storing the working matrix and, for solution bases, by storing O(n·nullity) extra numbers.

Code Examples

Rank over the reals with partial pivoting (RREF) and epsilon
1#include <bits/stdc++.h>
2using namespace std;
3
4// Compute rank of an m x n real matrix using RREF with partial pivoting.
5// Numerical stability: choose largest absolute pivot; treat |x| < eps as zero.
6int rank_real(vector<vector<double>> A, double eps = 1e-9) {
7 int m = (int)A.size();
8 if (m == 0) return 0;
9 int n = (int)A[0].size();
10 int r = 0; // current pivot row, also counts pivots
11 for (int c = 0; c < n && r < m; ++c) {
12 // Find pivot row with max |A[i][c]| for i >= r
13 int piv = r;
14 for (int i = r + 1; i < m; ++i) {
15 if (fabs(A[i][c]) > fabs(A[piv][c])) piv = i;
16 }
17 if (fabs(A[piv][c]) < eps) continue; // no pivot in this column
18 swap(A[piv], A[r]);
19 // Normalize pivot row so pivot becomes 1
20 double div = A[r][c];
21 for (int j = c; j < n; ++j) A[r][j] /= div;
22 // Eliminate this column from all other rows
23 for (int i = 0; i < m; ++i) if (i != r) {
24 double factor = A[i][c];
25 if (fabs(factor) < eps) continue;
26 for (int j = c; j < n; ++j) A[i][j] -= factor * A[r][j];
27 }
28 ++r; // found a pivot
29 }
30 return r;
31}
32
33int main() {
34 ios::sync_with_stdio(false);
35 cin.tie(nullptr);
36
37 int m, n;
38 // Example input: first line m n, followed by m lines each with n entries.
39 // 3 4
40 // 1 2 3 4
41 // 2 4 6 8
42 // 0 1 0 1
43 if (!(cin >> m >> n)) {
44 // Demo fallback
45 vector<vector<double>> A = {{1,2,3,4},{2,4,6,8},{0,1,0,1}};
46 cout << rank_real(A) << "\n"; // Expected rank: 2
47 return 0;
48 }
49 vector<vector<double>> A(m, vector<double>(n));
50 for (int i = 0; i < m; ++i)
51 for (int j = 0; j < n; ++j)
52 cin >> A[i][j];
53 cout << rank_real(A) << "\n";
54 return 0;
55}
56

Performs reduced row-echelon form with partial pivoting and counts pivots. The epsilon prevents treating tiny round-off noise as a nonzero pivot. The returned count equals the matrix rank and the dimension of both the column and row spaces.

Time: O(min(m, n) * m * n) (O(n^3) for square matrices)Space: O(m * n)
Rank modulo a prime p (Gaussian elimination over GF(p))
1#include <bits/stdc++.h>
2using namespace std;
3using int64 = long long;
4
5int64 mod_pow(int64 a, int64 e, int64 p) {
6 int64 r = 1 % p;
7 a %= p;
8 while (e) {
9 if (e & 1) r = (__int128)r * a % p;
10 a = (__int128)a * a % p;
11 e >>= 1;
12 }
13 return r;
14}
15
16int64 mod_inv(int64 a, int64 p) {
17 // p is prime: a^(p-2) mod p
18 a %= p; if (a < 0) a += p;
19 return mod_pow(a, p - 2, p);
20}
21
22int rank_mod_prime(vector<vector<int64>> A, int64 p) {
23 int m = (int)A.size();
24 if (m == 0) return 0;
25 int n = (int)A[0].size();
26 int r = 0;
27 for (int c = 0; c < n && r < m; ++c) {
28 int piv = -1;
29 for (int i = r; i < m; ++i) if (A[i][c] % p != 0) { piv = i; break; }
30 if (piv == -1) continue;
31 swap(A[piv], A[r]);
32 int64 inv = mod_inv(A[r][c], p);
33 // Normalize pivot row
34 for (int j = c; j < n; ++j) {
35 A[r][j] = (__int128)A[r][j] * inv % p;
36 }
37 // Eliminate this column from other rows
38 for (int i = 0; i < m; ++i) if (i != r) {
39 int64 factor = A[i][c] % p; if (factor < 0) factor += p;
40 if (factor == 0) continue;
41 for (int j = c; j < n; ++j) {
42 A[i][j] = (A[i][j] - (__int128)factor * A[r][j]) % p;
43 if (A[i][j] < 0) A[i][j] += p;
44 }
45 }
46 ++r;
47 }
48 return r;
49}
50
51int main() {
52 ios::sync_with_stdio(false);
53 cin.tie(nullptr);
54
55 int m, n; long long p;
56 // Input: m n p, then m lines each with n entries modulo p.
57 // Example: 2 3 7 \n 1 2 3 \n 2 4 6 -> rank is 1 over GF(7)
58 if (!(cin >> m >> n >> p)) {
59 vector<vector<int64>> A = {{1,2,3},{2,4,6}}; p = 7;
60 cout << rank_mod_prime(A, p) << "\n"; // 1
61 return 0;
62 }
63 vector<vector<int64>> A(m, vector<int64>(n));
64 for (int i = 0; i < m; ++i)
65 for (int j = 0; j < n; ++j)
66 cin >> A[i][j];
67 int r = rank_mod_prime(A, p);
68 cout << r << "\n";
69 return 0;
70}
71

Computes rank over the finite field GF(p) with modular inverses via Fermat’s little theorem (since p is prime). This is crucial in competitive programming problems involving modular linear algebra and independence checks modulo p.

Time: O(min(m, n) * m * n) field ops; modular inverses cost O(log p) per pivotSpace: O(m * n)
GF(2) XOR basis (rank over binary field) with queries
1#include <bits/stdc++.h>
2using namespace std;
3
4// Maintain a linear basis over GF(2) for 64-bit integers.
5// Insertion reduces the vector; if it becomes zero, it's dependent; otherwise it adds a new basis vector.
6struct XorBasis64 {
7 uint64_t b[64]{}; // b[i] has leading bit i
8 int sz = 0;
9
10 void insert(uint64_t x) {
11 for (int i = 63; i >= 0; --i) if (x & (1ull << i)) {
12 if (!b[i]) { b[i] = x; ++sz; return; }
13 x ^= b[i];
14 }
15 // if x becomes 0, it's dependent and ignored
16 }
17
18 bool canRepresent(uint64_t x) const {
19 for (int i = 63; i >= 0; --i) if (x & (1ull << i)) {
20 if (!b[i]) return false;
21 x ^= b[i];
22 }
23 return true; // reduced to 0
24 }
25
26 // Maximize x XOR y over all y in span
27 uint64_t maximize(uint64_t x = 0) const {
28 uint64_t res = x;
29 for (int i = 63; i >= 0; --i) if (b[i]) {
30 res = max(res, res ^ b[i]);
31 }
32 return res;
33 }
34};
35
36int main() {
37 ios::sync_with_stdio(false);
38 cin.tie(nullptr);
39
40 int n; // number of vectors
41 // Example: n then n numbers; then q queries: type t and value v
42 // t=1: ask if v is representable; t=2: print max XOR with v; t=3: print basis size (rank)
43 if (!(cin >> n)) {
44 XorBasis64 B; B.insert(5); B.insert(7); B.insert(2); // {0101, 0111, 0010}
45 cout << B.sz << "\n"; // rank over GF(2)
46 cout << B.canRepresent(5 ^ 7) << "\n"; // 1 (true)
47 cout << B.maximize(0) << "\n"; // maximum XOR attainable
48 return 0;
49 }
50 XorBasis64 B;
51 for (int i = 0; i < n; ++i) { uint64_t x; cin >> x; B.insert(x); }
52 int q; cin >> q;
53 while (q--) {
54 int t; uint64_t v; cin >> t >> v;
55 if (t == 1) cout << (B.canRepresent(v) ? "YES" : "NO") << "\n";
56 else if (t == 2) cout << B.maximize(v) << "\n";
57 else if (t == 3) cout << B.sz << "\n";
58 }
59 return 0;
60}
61

Implements a 64-bit XOR basis that represents rank computation over GF(2). Inserting a number increases the basis size only if it adds a new independent direction; the basis size is the rank. The structure also supports membership queries and maximum XOR queries, common in CF tasks.

Time: O(64^2) per insertion in the worst case; O(64) per querySpace: O(64)
Solve Ax = b, check consistency, and extract a null-space basis (reals)
1#include <bits/stdc++.h>
2using namespace std;
3
4struct RREFResult {
5 int rank;
6 vector<int> pivot_col; // pivot column indices in order of pivot rows
7 vector<vector<double>> M; // RREF of augmented matrix [A|b]
8 bool consistent;
9};
10
11RREFResult rref_augmented(vector<vector<double>> Ab, double eps = 1e-9) {
12 int m = (int)Ab.size();
13 int n_aug = (int)Ab[0].size(); // n + 1 (augmented)
14 int n = n_aug - 1;
15 int r = 0; // pivot row
16 vector<int> pivcol;
17 for (int c = 0; c < n && r < m; ++c) {
18 int piv = r;
19 for (int i = r + 1; i < m; ++i)
20 if (fabs(Ab[i][c]) > fabs(Ab[piv][c])) piv = i;
21 if (fabs(Ab[piv][c]) < eps) continue;
22 swap(Ab[piv], Ab[r]);
23 double div = Ab[r][c];
24 for (int j = c; j < n_aug; ++j) Ab[r][j] /= div;
25 for (int i = 0; i < m; ++i) if (i != r) {
26 double f = Ab[i][c]; if (fabs(f) < eps) continue;
27 for (int j = c; j < n_aug; ++j) Ab[i][j] -= f * Ab[r][j];
28 }
29 pivcol.push_back(c);
30 ++r;
31 }
32 // Check consistency: any row with all zeros in A part but nonzero in b?
33 bool consistent = true;
34 for (int i = 0; i < m; ++i) {
35 bool allzero = true;
36 for (int j = 0; j < n; ++j) allzero &= fabs(Ab[i][j]) < eps;
37 if (allzero && fabs(Ab[i][n]) > eps) { consistent = false; break; }
38 }
39 return {r, pivcol, Ab, consistent};
40}
41
42int main() {
43 ios::sync_with_stdio(false);
44 cin.tie(nullptr);
45
46 int m, n;
47 // Input: m n, then A (m x n), then b (m entries)
48 // Example:
49 // 2 3
50 // 1 2 0
51 // 0 1 1
52 // 3 2
53 if (!(cin >> m >> n)) {
54 vector<vector<double>> A = {{1,2,0},{0,1,1}}; vector<double> b = {3,2};
55 vector<vector<double>> Ab(m, vector<double>(n+1));
56 for (int i = 0; i < m; ++i) { for (int j = 0; j < n; ++j) Ab[i][j] = A[i][j]; Ab[i][n] = b[i]; }
57 auto res = rref_augmented(Ab);
58 cout << (res.consistent ? "CONSISTENT\n" : "INCONSISTENT\n");
59 cout << "rank(A) = " << res.rank << "\n";
60 if (res.consistent) {
61 // Particular solution: set all free vars = 0, read pivot vars from RHS
62 vector<double> x(n, 0.0);
63 for (int i = 0; i < (int)res.pivot_col.size(); ++i) {
64 int c = res.pivot_col[i];
65 x[c] = res.M[i][n];
66 }
67 cout << "one solution:";
68 for (int j = 0; j < n; ++j) cout << ' ' << fixed << setprecision(9) << x[j];
69 cout << "\n";
70 // Null space basis: one vector per free variable f
71 vector<int> is_pivot(n, 0);
72 for (int c : res.pivot_col) is_pivot[c] = 1;
73 vector<int> free_cols;
74 for (int j = 0; j < n; ++j) if (!is_pivot[j]) free_cols.push_back(j);
75 cout << "nullity = " << free_cols.size() << "\n";
76 for (int idx = 0; idx < (int)free_cols.size(); ++idx) {
77 int f = free_cols[idx];
78 vector<double> v(n, 0.0);
79 v[f] = 1.0; // set free var = 1
80 // For each pivot row i with pivot at c, x_c = -sum_j (row[i][j] * x_j) when j is free
81 for (int i = 0; i < (int)res.pivot_col.size(); ++i) {
82 int c = res.pivot_col[i];
83 v[c] = -res.M[i][f];
84 }
85 cout << "null vector " << idx << ":";
86 for (int j = 0; j < n; ++j) cout << ' ' << fixed << setprecision(9) << v[j];
87 cout << "\n";
88 }
89 }
90 return 0;
91 }
92
93 vector<vector<double>> Ab(m, vector<double>(n+1));
94 for (int i = 0; i < m; ++i)
95 for (int j = 0; j < n; ++j)
96 cin >> Ab[i][j];
97 for (int i = 0; i < m; ++i) cin >> Ab[i][n];
98
99 auto res = rref_augmented(Ab);
100 cout << (res.consistent ? "CONSISTENT\n" : "INCONSISTENT\n");
101 cout << "rank(A) = " << res.rank << "\n";
102 if (res.consistent) {
103 vector<double> x(n, 0.0);
104 for (int i = 0; i < (int)res.pivot_col.size(); ++i) x[res.pivot_col[i]] = res.M[i][n];
105 cout << "one solution:";
106 for (int j = 0; j < n; ++j) cout << ' ' << fixed << setprecision(9) << x[j];
107 cout << "\n";
108 vector<int> is_pivot(n, 0);
109 for (int c : res.pivot_col) is_pivot[c] = 1;
110 vector<int> free_cols; for (int j = 0; j < n; ++j) if (!is_pivot[j]) free_cols.push_back(j);
111 cout << "nullity = " << free_cols.size() << "\n";
112 for (int idx = 0; idx < (int)free_cols.size(); ++idx) {
113 int f = free_cols[idx];
114 vector<double> v(n, 0.0); v[f] = 1.0;
115 for (int i = 0; i < (int)res.pivot_col.size(); ++i) v[res.pivot_col[i]] = -res.M[i][f];
116 cout << "null vector " << idx << ":";
117 for (int j = 0; j < n; ++j) cout << ' ' << fixed << setprecision(9) << v[j];
118 cout << "\n";
119 }
120 }
121 return 0;
122}
123

Reduces the augmented matrix [A|b] to RREF with partial pivoting, checks consistency using Rouché–Capelli, constructs one particular solution (setting all free variables to zero), and outputs a basis for the null space (one vector per free variable). The nullity equals the number of free variables.

Time: O(min(m, n) * m * n) (RREF on [A|b])Space: O(m * (n + 1))