Groups
Matrix factorizations rewrite a matrix into simpler building blocks (triangular or orthogonal) that make solving and analyzing linear systems much easier.
A matrix inverse undoes the effect of a linear transformation, just like dividing by a number undoes multiplication.