<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" lang="" xml:lang="">
<head>
<meta charset="utf-8" />
<meta name="generator" content="pandoc" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
<meta name="dcterms.date" content="2023-07-29" />
<title>Linear Algebra, in a nutshell</title>
<style>
html {
color: #1a1a1a;
background-color: #fdfdfd;
}
body {
margin: 0 auto;
max-width: 36em;
padding-left: 50px;
padding-right: 50px;
padding-top: 50px;
padding-bottom: 50px;
hyphens: auto;
overflow-wrap: break-word;
text-rendering: optimizeLegibility;
font-kerning: normal;
}
@media (max-width: 600px) {
body {
font-size: 0.9em;
padding: 12px;
}
h1 {
font-size: 1.8em;
}
}
@media print {
html {
background-color: white;
}
body {
background-color: transparent;
color: black;
font-size: 12pt;
}
p, h2, h3 {
orphans: 3;
widows: 3;
}
h2, h3, h4 {
page-break-after: avoid;
}
}
p {
margin: 1em 0;
}
a {
color: #1a1a1a;
}
a:visited {
color: #1a1a1a;
}
img {
max-width: 100%;
}
svg {
height: auto;
max-width: 100%;
}
h1, h2, h3, h4, h5, h6 {
margin-top: 1.4em;
}
h5, h6 {
font-size: 1em;
font-style: italic;
}
h6 {
font-weight: normal;
}
ol, ul {
padding-left: 1.7em;
margin-top: 1em;
}
li > ol, li > ul {
margin-top: 0;
}
blockquote {
margin: 1em 0 1em 1.7em;
padding-left: 1em;
border-left: 2px solid #e6e6e6;
color: #606060;
}
code {
font-family: Menlo, Monaco, Consolas, 'Lucida Console', monospace;
font-size: 85%;
margin: 0;
hyphens: manual;
}
pre {
margin: 1em 0;
overflow: auto;
}
pre code {
padding: 0;
overflow: visible;
overflow-wrap: normal;
}
.sourceCode {
background-color: transparent;
overflow: visible;
}
hr {
border: none;
border-top: 1px solid #1a1a1a;
height: 1px;
margin: 1em 0;
}
table {
margin: 1em 0;
border-collapse: collapse;
width: 100%;
overflow-x: auto;
display: block;
font-variant-numeric: lining-nums tabular-nums;
}
table caption {
margin-bottom: 0.75em;
}
tbody {
margin-top: 0.5em;
border-top: 1px solid #1a1a1a;
border-bottom: 1px solid #1a1a1a;
}
th {
border-top: 1px solid #1a1a1a;
padding: 0.25em 0.5em 0.25em 0.5em;
}
td {
padding: 0.125em 0.5em 0.25em 0.5em;
}
header {
margin-bottom: 4em;
text-align: center;
}
#TOC li {
list-style: none;
}
#TOC ul {
padding-left: 1.3em;
}
#TOC > ul {
padding-left: 0;
}
#TOC a:not(:hover) {
text-decoration: none;
}
code{white-space: pre-wrap;}
span.smallcaps{font-variant: small-caps;}
div.columns{display: flex; gap: min(4vw, 1.5em);}
div.column{flex: auto; overflow-x: auto;}
div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}
/* The extra [class] is a hack that increases specificity enough to
override a similar rule in reveal.js */
ul.task-list[class]{list-style: none;}
ul.task-list li input[type="checkbox"] {
font-size: inherit;
width: 0.8em;
margin: 0 0.8em 0.2em -1.6em;
vertical-align: middle;
}
</style>
</head>
<body>
<header id="title-block-header">
<h1 class="title">Linear Algebra, in a nutshell</h1>
<p class="date">2023-07-29</p>
</header>
<h2 id="vectors">Vectors</h2>
<p>There are these things called vectors. They don’t exist in isolation.
Rather, they’re defined as parts of <em>vector spaces</em>. A vector
space has objects called <em>vectors</em>
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>u</mi><annotation encoding="application/x-tex">u</annotation></semantics></math>,
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>v</mi><annotation encoding="application/x-tex">v</annotation></semantics></math>
and <em>scalars</em>
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>k</mi><annotation encoding="application/x-tex">k</annotation></semantics></math>
such that
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>u</mi><mo>+</mo><mi>v</mi></mrow><annotation encoding="application/x-tex">u+v</annotation></semantics></math>
is a vector and
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi><mi>v</mi></mrow><annotation encoding="application/x-tex">kv</annotation></semantics></math>
is a scalar. Vector addition and scalar multiplication behave as you’d
expect: it commutes, associates, distributes, etc. Don’t think too hard
about what vectors <em>are</em>, just what they <em>do</em>.</p>
<p>The canonical example of a vector is a list of real or complex
numbers. These vectors exist in
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>â„</mi><mi>n</mi></msup><annotation encoding="application/x-tex">\mathbb{R}^n</annotation></semantics></math>
or
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>â„‚</mi><mi>n</mi></msup><annotation encoding="application/x-tex">\mathbb{C}^n</annotation></semantics></math>.
We will be focusing on these vectors soon, but for now we work with
generic vector spaces.</p>
<p>We say a set of vectors
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false" form="prefix">{</mo><msub><mi>v</mi><mi>i</mi></msub><mo stretchy="false" form="postfix">}</mo></mrow><annotation encoding="application/x-tex">\{v_i\}</annotation></semantics></math>
is <em>linearly independent</em> if the only solution to
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo>∑</mo><msub><mi>a</mi><mi>i</mi></msub><msub><mi>v</mi><mi>i</mi></msub><mo>=</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">\sum a_iv_i = 0</annotation></semantics></math>
is
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>a</mi><mi>i</mi></msub><mo>=</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">a_i=0</annotation></semantics></math>
for all
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>a</mi><mi>i</mi></msub><annotation encoding="application/x-tex">a_i</annotation></semantics></math>.
We say a set of vectors <em>spans</em> a vector space
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>V</mi><annotation encoding="application/x-tex">V</annotation></semantics></math>
if for any vector
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>v</mi><mo>∈</mo><mi>V</mi></mrow><annotation encoding="application/x-tex">v\in V</annotation></semantics></math>,
there is some solution to
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo>∑</mo><msub><mi>a</mi><mi>i</mi></msub><msub><mi>v</mi><mi>i</mi></msub><mo>=</mo><mi>v</mi></mrow><annotation encoding="application/x-tex">\sum a_iv_i = v</annotation></semantics></math>.
If
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false" form="prefix">{</mo><msub><mi>v</mi><mi>i</mi></msub><mo stretchy="false" form="postfix">}</mo></mrow><annotation encoding="application/x-tex">\{v_i\}</annotation></semantics></math>
is linearly independent and spans
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>V</mi><annotation encoding="application/x-tex">V</annotation></semantics></math>,
then we say
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false" form="prefix">{</mo><msub><mi>v</mi><mi>i</mi></msub><mo stretchy="false" form="postfix">}</mo></mrow><annotation encoding="application/x-tex">\{v_i\}</annotation></semantics></math>
is a <em>basis</em> of
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>V</mi><annotation encoding="application/x-tex">V</annotation></semantics></math>.</p>
<p>All bases of a vector have the same length (i.e. the same number of
vectors). Therefore we define the <em>dimension</em> of a vector space
to be the length of the basis.</p>
<p>A <em>subspace</em>
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>U</mi><annotation encoding="application/x-tex">U</annotation></semantics></math>
of a vector space
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>V</mi><annotation encoding="application/x-tex">V</annotation></semantics></math>
is a vector space whose elements are all in
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>V</mi><annotation encoding="application/x-tex">V</annotation></semantics></math>
and whose addition/multiplication operations are identical. That is, if
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>u</mi><mo>+</mo><mi>v</mi><mo>=</mo><mi>w</mi></mrow><annotation encoding="application/x-tex">u+v=w</annotation></semantics></math>
under
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>U</mi><annotation encoding="application/x-tex">U</annotation></semantics></math>,
then
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>u</mi><mo>+</mo><mi>v</mi><mo>=</mo><mi>w</mi></mrow><annotation encoding="application/x-tex">u+v=w</annotation></semantics></math>
under
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>V</mi><annotation encoding="application/x-tex">V</annotation></semantics></math>
as well, etc.</p>
<h2 id="linear-maps">Linear Maps</h2>
<p>A linear map
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>T</mi><annotation encoding="application/x-tex">T</annotation></semantics></math>
is a function
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>V</mi><mo>→</mo><mi>W</mi></mrow><annotation encoding="application/x-tex">V\to W</annotation></semantics></math>
such that
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>T</mi><mo stretchy="false" form="prefix">(</mo><mi>u</mi><mo>+</mo><mi>v</mi><mo stretchy="false" form="postfix">)</mo><mo>=</mo><mi>T</mi><mi>u</mi><mo>+</mo><mi>T</mi><mi>v</mi></mrow><annotation encoding="application/x-tex">T(u+v)=Tu+Tv</annotation></semantics></math>
and
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>T</mi><mo stretchy="false" form="prefix">(</mo><mi>a</mi><mi>v</mi><mo stretchy="false" form="postfix">)</mo><mo>=</mo><mi>a</mi><mi>T</mi><mi>v</mi></mrow><annotation encoding="application/x-tex">T(av) = aTv</annotation></semantics></math>
for all vectors
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>u</mi><annotation encoding="application/x-tex">u</annotation></semantics></math>,
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>v</mi><annotation encoding="application/x-tex">v</annotation></semantics></math>
and all scalars
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>a</mi><annotation encoding="application/x-tex">a</annotation></semantics></math>.
(<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>V</mi><annotation encoding="application/x-tex">V</annotation></semantics></math>
and
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>W</mi><annotation encoding="application/x-tex">W</annotation></semantics></math>
are vector spaces.)</p>
<p>The most boring map is
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>I</mi><annotation encoding="application/x-tex">I</annotation></semantics></math>,
the identity map. It is the function
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>V</mi><mo>→</mo><mi>V</mi></mrow><annotation encoding="application/x-tex">V\to V</annotation></semantics></math>
such that
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>T</mi><mi>v</mi><mo>=</mo><mi>v</mi></mrow><annotation encoding="application/x-tex">Tv = v</annotation></semantics></math>.</p>
<p>The <em>null space</em> of a linear map is the set of vectors
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>v</mi><annotation encoding="application/x-tex">v</annotation></semantics></math>
such that
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>T</mi><mi>v</mi><mo>=</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">Tv=0</annotation></semantics></math>.
As the term implies, the null space is a vector space itself. The
<em>range</em> of a linear map is the set of vectors
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>v</mi><annotation encoding="application/x-tex">v</annotation></semantics></math>
such that there exists some vector
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>u</mi><annotation encoding="application/x-tex">u</annotation></semantics></math>
satisfying
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>T</mi><mi>u</mi><mo>=</mo><mi>v</mi></mrow><annotation encoding="application/x-tex">Tu=v</annotation></semantics></math>.
It is also a vector space. In other words, the range is the set of
outputs possible from
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>T</mi><annotation encoding="application/x-tex">T</annotation></semantics></math>.
(Take note that
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mrow><mtext mathvariant="normal">null </mtext><mspace width="0.333em"></mspace></mrow><mi>T</mi><mo>⊆</mo><mi>V</mi></mrow><annotation encoding="application/x-tex">\text{null } T \subseteq V</annotation></semantics></math>
and
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mrow><mtext mathvariant="normal">range </mtext><mspace width="0.333em"></mspace></mrow><mi>T</mi><mo>⊆</mo><mi>W</mi></mrow><annotation encoding="application/x-tex">\text{range } T \subseteq W</annotation></semantics></math>.)
The <em>Rank-Nullity Theorem</em> states
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mrow><mtext mathvariant="normal">dim </mtext><mspace width="0.333em"></mspace></mrow><mrow><mtext mathvariant="normal">null </mtext><mspace width="0.333em"></mspace></mrow><mi>T</mi><mo>+</mo><mrow><mtext mathvariant="normal">dim </mtext><mspace width="0.333em"></mspace></mrow><mrow><mtext mathvariant="normal">range </mtext><mspace width="0.333em"></mspace></mrow><mi>T</mi><mo>=</mo><mrow><mtext mathvariant="normal">dim </mtext><mspace width="0.333em"></mspace></mrow><mi>V</mi></mrow><annotation encoding="application/x-tex">\text{dim } \text{null } T + \text{dim } \text{range } T = \text{dim } V</annotation></semantics></math>.</p>
<p>Also, linear maps can be injective, surjective, and invertible.
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>T</mi><annotation encoding="application/x-tex">T</annotation></semantics></math>
is injective if
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>a</mi><mo>=</mo><mi>b</mi><mo>⇔</mo><mi>T</mi><mi>a</mi><mo>=</mo><mi>T</mi><mi>b</mi></mrow><annotation encoding="application/x-tex">a = b \iff Ta = Tb</annotation></semantics></math>.
(<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>T</mi><annotation encoding="application/x-tex">T</annotation></semantics></math>
injective is equivalent to
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mrow><mtext mathvariant="normal">null </mtext><mspace width="0.333em"></mspace></mrow><mi>T</mi><mo>=</mo><mo stretchy="false" form="prefix">{</mo><mn>0</mn><mo stretchy="false" form="postfix">}</mo></mrow><annotation encoding="application/x-tex">\text{null } T = \{0\}</annotation></semantics></math>.)
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>T</mi><annotation encoding="application/x-tex">T</annotation></semantics></math>
is surjective if
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mrow><mtext mathvariant="normal">range </mtext><mspace width="0.333em"></mspace></mrow><mi>T</mi><mo>=</mo><mi>W</mi></mrow><annotation encoding="application/x-tex">\text{range } T = W</annotation></semantics></math>.
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>T</mi><annotation encoding="application/x-tex">T</annotation></semantics></math>
is bijective if it is both injective and surjective. And
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>T</mi><annotation encoding="application/x-tex">T</annotation></semantics></math>
is invertible if
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>T</mi><msup><mi>T</mi><mrow><mi>−</mi><mn>1</mn></mrow></msup><mo>=</mo><msup><mi>T</mi><mrow><mi>−</mi><mn>1</mn></mrow></msup><mi>T</mi><mo>=</mo><mi>I</mi></mrow><annotation encoding="application/x-tex">TT^{-1} = T^{-1}T = I</annotation></semantics></math>.
Obviously,
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>T</mi><annotation encoding="application/x-tex">T</annotation></semantics></math>
must be of the form
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>V</mi><mo>→</mo><mi>V</mi></mrow><annotation encoding="application/x-tex">V\to V</annotation></semantics></math>,
i.e. <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>V</mi><mo>=</mo><mi>W</mi></mrow><annotation encoding="application/x-tex">V=W</annotation></semantics></math>,
for
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>T</mi><annotation encoding="application/x-tex">T</annotation></semantics></math>
to be invertible.</p>
<p>A <em>linear endomorphism</em> is a map of the form
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>V</mi><mo>→</mo><mi>V</mi></mrow><annotation encoding="application/x-tex">V\to V</annotation></semantics></math>.
In other words, the domain and codomain are identical; this is the
definition of an endomorphism. In an endomorphism, injectivity,
surjectivity, and invertibility are all identical.</p>
<p>Now we discuss <em>invariant subspaces</em> on linear endomorphisms.
A subspace
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>U</mi><annotation encoding="application/x-tex">U</annotation></semantics></math>
is invariant under a transformation
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>T</mi><annotation encoding="application/x-tex">T</annotation></semantics></math>
if
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>T</mi><mi>u</mi><mo>∈</mo><mi>U</mi></mrow><annotation encoding="application/x-tex">Tu \in U</annotation></semantics></math>
for all
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>u</mi><mo>∈</mo><mi>U</mi></mrow><annotation encoding="application/x-tex">u \in U</annotation></semantics></math>.
You can decompose a vector space into invariant subspaces.</p>
<p>The most useful kind of invariant subspace is that with dimension
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mn>1</mn><annotation encoding="application/x-tex">1</annotation></semantics></math>.
These subspaces are essentially equivalent to <em>eigenvectors</em>. We
say
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>v</mi><annotation encoding="application/x-tex">v</annotation></semantics></math>
is an eigenvector if
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>T</mi><mi>v</mi><mo>=</mo><mi>λ</mi><mi>v</mi></mrow><annotation encoding="application/x-tex">Tv = \lambda v</annotation></semantics></math>
for some scalar
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>v</mi><annotation encoding="application/x-tex">v</annotation></semantics></math>.
Also, we call
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>λ</mi><annotation encoding="application/x-tex">\lambda</annotation></semantics></math>
the associated <em>eigenvalue</em>. If
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>v</mi><annotation encoding="application/x-tex">v</annotation></semantics></math>
is an eigenvector, then
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false" form="prefix">{</mo><mi>a</mi><mi>v</mi><mo stretchy="false" form="postfix">}</mo></mrow><annotation encoding="application/x-tex">\{av\}</annotation></semantics></math>
is an invariant subspace with dimension
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mn>1</mn><annotation encoding="application/x-tex">1</annotation></semantics></math>.</p>
<p>A set of eigenvectors in distinct
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mrow><mtext mathvariant="normal">dim </mtext><mspace width="0.333em"></mspace></mrow><mn>1</mn></mrow><annotation encoding="application/x-tex">\text{dim } 1</annotation></semantics></math>
invariant subspaces is linearly independent.</p>
<p>All complex vector spaces have an eigenvalue. All real vector spaces
with odd dimension have an eigenvalue. The former is essentially
equivalent to the Fundamental Theorem of Algebra, and the latter is
equivalent to the fact that all real polynomials with odd degree have a
root.</p>
<h2 id="inner-product-spaces">Inner-Product Spaces</h2>
<p>Up until now any arbitrary vector space sufficed: the scalars could
come from any field. Now we must specifically deal with either the real
or complex vector spaces
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>â„</mi><mi>n</mi></msup><annotation encoding="application/x-tex">\mathbb{R}^n</annotation></semantics></math>
and
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>â„‚</mi><mi>n</mi></msup><annotation encoding="application/x-tex">\mathbb{C}^n</annotation></semantics></math>.</p>
<p>An inner product
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false" form="prefix">⟨</mo><mi>u</mi><mo>,</mo><mi>v</mi><mo stretchy="false" form="postfix">⟩</mo></mrow><annotation encoding="application/x-tex">\langle u, v\rangle</annotation></semantics></math>
satisfies the following:</p>
<ul>
<li><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false" form="prefix">⟨</mo><mi>v</mi><mo>,</mo><mi>v</mi><mo stretchy="false" form="postfix">⟩</mo><mo>≥</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">\langle v, v\rangle \geq 0</annotation></semantics></math>,
with equality iff
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>v</mi><mo>=</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">v=0</annotation></semantics></math></li>
<li><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false" form="prefix">⟨</mo><mi>u</mi><mo>+</mo><mi>v</mi><mo>,</mo><mi>w</mi><mo stretchy="false" form="postfix">⟩</mo><mo>=</mo><mo stretchy="false" form="prefix">⟨</mo><mi>u</mi><mo>,</mo><mi>w</mi><mo stretchy="false" form="postfix">⟩</mo><mo>+</mo><mo stretchy="false" form="prefix">⟨</mo><mi>v</mi><mo>,</mo><mi>w</mi><mo stretchy="false" form="postfix">⟩</mo></mrow><annotation encoding="application/x-tex">\langle u+v, w\rangle = \langle u, w\rangle + \langle v, w\rangle</annotation></semantics></math></li>
<li><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false" form="prefix">⟨</mo><mi>a</mi><mi>v</mi><mo>,</mo><mi>w</mi><mo stretchy="false" form="postfix">⟩</mo><mo>=</mo><mi>a</mi><mo stretchy="false" form="prefix">⟨</mo><mi>v</mi><mo>,</mo><mi>w</mi><mo stretchy="false" form="postfix">⟩</mo></mrow><annotation encoding="application/x-tex">\langle av, w\rangle = a\langle v, w\rangle</annotation></semantics></math></li>
<li><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false" form="prefix">⟨</mo><mi>v</mi><mo>,</mo><mi>w</mi><mo stretchy="false" form="postfix">⟩</mo><mo>=</mo><mover><mrow><mo stretchy="false" form="prefix">⟨</mo><mi>w</mi><mo>,</mo><mi>v</mi><mo stretchy="false" form="postfix">⟩</mo></mrow><mo accent="true">¯</mo></mover></mrow><annotation encoding="application/x-tex">\langle v, w\rangle = \overline{\langle w, v\rangle}</annotation></semantics></math></li>
</ul>
<p>(To clarify,
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mover><mi>z</mi><mo accent="true">¯</mo></mover><annotation encoding="application/x-tex">\overline{z}</annotation></semantics></math>
is the complex conjugate of
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>z</mi><annotation encoding="application/x-tex">z</annotation></semantics></math>.)</p>
<p>These are our <em>axioms</em>, but one can also deduce that
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false" form="prefix">⟨</mo><mi>u</mi><mo>,</mo><mi>v</mi><mo>+</mo><mi>w</mi><mo stretchy="false" form="postfix">⟩</mo><mo>=</mo><mo stretchy="false" form="prefix">⟨</mo><mi>u</mi><mo>,</mo><mi>v</mi><mo stretchy="false" form="postfix">⟩</mo><mo>+</mo><mo stretchy="false" form="prefix">⟨</mo><mi>u</mi><mo>,</mo><mi>w</mi><mo stretchy="false" form="postfix">⟩</mo></mrow><annotation encoding="application/x-tex">\langle u, v+w\rangle = \langle u, v\rangle + \langle u, w\rangle</annotation></semantics></math>
and
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false" form="prefix">⟨</mo><mi>v</mi><mo>,</mo><mi>a</mi><mi>w</mi><mo stretchy="false" form="postfix">⟩</mo><mo>=</mo><mover><mi>a</mi><mo accent="true">¯</mo></mover><mo stretchy="false" form="prefix">⟨</mo><mi>v</mi><mo>,</mo><mi>w</mi><mo stretchy="false" form="postfix">⟩</mo></mrow><annotation encoding="application/x-tex">\langle v, aw\rangle = \overline{a}\langle v, w\rangle</annotation></semantics></math>.</p>
<p>The standard example of an inner product is the dot product. Note
that the complex dot product is not quite what you’d expect;
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false" form="prefix">⟨</mo><mo stretchy="false" form="prefix">(</mo><msub><mi>w</mi><mi>i</mi></msub><mo stretchy="false" form="postfix">)</mo><mo>,</mo><mo stretchy="false" form="prefix">(</mo><msub><mi>z</mi><mi>i</mi></msub><mo stretchy="false" form="postfix">)</mo><mo stretchy="false" form="postfix">⟩</mo><mo>=</mo><mo>∑</mo><msub><mi>w</mi><mi>i</mi></msub><mover><msub><mi>z</mi><mi>i</mi></msub><mo accent="true">¯</mo></mover></mrow><annotation encoding="application/x-tex">\langle (w_i), (z_i)\rangle = \sum w_i\overline{z_i}</annotation></semantics></math>.
The motivation is so that
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false" form="prefix">⟨</mo><mi>z</mi><mo>,</mo><mi>z</mi><mo stretchy="false" form="postfix">⟩</mo><mo>=</mo><msqrt><mrow><mo>∑</mo><mo stretchy="false" form="prefix">|</mo><msub><mi>z</mi><mi>i</mi></msub><msup><mo stretchy="false" form="prefix">|</mo><mn>2</mn></msup></mrow></msqrt></mrow><annotation encoding="application/x-tex">\langle z, z\rangle = \sqrt{\sum |z_i|^2}</annotation></semantics></math>.</p>
<p>Also, the <em>norm</em>
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false" form="prefix">|</mo><mo stretchy="false" form="prefix">|</mo><mi>v</mi><mo stretchy="false" form="prefix">|</mo><mo stretchy="false" form="prefix">|</mo></mrow><annotation encoding="application/x-tex">||v||</annotation></semantics></math>
of
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>v</mi><annotation encoding="application/x-tex">v</annotation></semantics></math>
is
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msqrt><mrow><mo stretchy="false" form="prefix">⟨</mo><mi>v</mi><mo>,</mo><mi>v</mi><mo stretchy="false" form="postfix">⟩</mo></mrow></msqrt><annotation encoding="application/x-tex">\sqrt{\langle v, v\rangle}</annotation></semantics></math>.</p>
<p>Some (in)equalities you might expect from dealing with the dot
product:</p>
<ul>
<li>Pythagorean Theorem: If
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false" form="prefix">⟨</mo><mi>u</mi><mo>,</mo><mi>v</mi><mo stretchy="false" form="postfix">⟩</mo><mo>=</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">\langle u, v\rangle = 0</annotation></semantics></math>,
then
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false" form="prefix">|</mo><mo stretchy="false" form="prefix">|</mo><mi>u</mi><mo>+</mo><mi>v</mi><mo stretchy="false" form="prefix">|</mo><msup><mo stretchy="false" form="prefix">|</mo><mn>2</mn></msup><mo>=</mo><mo stretchy="false" form="prefix">|</mo><mo stretchy="false" form="prefix">|</mo><mi>u</mi><mo stretchy="false" form="prefix">|</mo><msup><mo stretchy="false" form="prefix">|</mo><mn>2</mn></msup><mo>+</mo><mo stretchy="false" form="prefix">|</mo><mo stretchy="false" form="prefix">|</mo><mi>v</mi><mo stretchy="false" form="prefix">|</mo><msup><mo stretchy="false" form="prefix">|</mo><mn>2</mn></msup></mrow><annotation encoding="application/x-tex">||u+v||^2 = ||u||^2+||v||^2</annotation></semantics></math>.</li>
<li>Cauchy-Schwarz:
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false" form="prefix">|</mo><mo stretchy="false" form="prefix">⟨</mo><mi>u</mi><mo>,</mo><mi>v</mi><mo stretchy="false" form="postfix">⟩</mo><mo stretchy="false" form="prefix">|</mo><mo>≤</mo><mo stretchy="false" form="prefix">|</mo><mo stretchy="false" form="prefix">|</mo><mi>u</mi><mo stretchy="false" form="prefix">|</mo><mo stretchy="false" form="prefix">|</mo><mo stretchy="false" form="prefix">|</mo><mo stretchy="false" form="prefix">|</mo><mi>v</mi><mo stretchy="false" form="prefix">|</mo><mo stretchy="false" form="prefix">|</mo></mrow><annotation encoding="application/x-tex">|\langle u, v\rangle| \leq ||u|| ||v||</annotation></semantics></math>,
with equality when one of
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>u</mi><annotation encoding="application/x-tex">u</annotation></semantics></math>,
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>v</mi><annotation encoding="application/x-tex">v</annotation></semantics></math>
is a multiple of the other.</li>
<li>Triangle Inequality:
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false" form="prefix">|</mo><mo stretchy="false" form="prefix">|</mo><mi>u</mi><mo>+</mo><mi>v</mi><mo stretchy="false" form="prefix">|</mo><mo stretchy="false" form="prefix">|</mo><mo>≤</mo><mo stretchy="false" form="prefix">|</mo><mo stretchy="false" form="prefix">|</mo><mi>u</mi><mo stretchy="false" form="prefix">|</mo><mo stretchy="false" form="prefix">|</mo><mi>+</mi><mo stretchy="false" form="prefix">|</mo><mo stretchy="false" form="prefix">|</mo><mi>v</mi><mo stretchy="false" form="prefix">|</mo><mo stretchy="false" form="prefix">|</mo></mrow><annotation encoding="application/x-tex">||u+v||\leq ||u||+||v||</annotation></semantics></math>,
with equality when one of
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>u</mi><annotation encoding="application/x-tex">u</annotation></semantics></math>,
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>v</mi><annotation encoding="application/x-tex">v</annotation></semantics></math>
is a non-negative multiple of the other.</li>
</ul>
<p>With the inner product defined, we can construct <em>orthonormal</em>
bases of a vector space. In an orthonormal basis,
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false" form="prefix">⟨</mo><msub><mi>e</mi><mi>i</mi></msub><mo>,</mo><msub><mi>e</mi><mi>j</mi></msub><mo stretchy="false" form="postfix">⟩</mo></mrow><annotation encoding="application/x-tex">\langle e_i, e_j\rangle</annotation></semantics></math>
is
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mn>1</mn><annotation encoding="application/x-tex">1</annotation></semantics></math>
if
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>i</mi><mo>=</mo><mi>j</mi></mrow><annotation encoding="application/x-tex">i=j</annotation></semantics></math>
and
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mn>0</mn><annotation encoding="application/x-tex">0</annotation></semantics></math>
if
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>i</mi><mo>≠</mo><mi>j</mi></mrow><annotation encoding="application/x-tex">i\neq j</annotation></semantics></math>.
An orthonormal basis always exists, and can be explicitly constructed
given some basis with the <em>Gram-Schmidt Procedure</em>.</p>
<h2 id="spectral-theorem">Spectral Theorem</h2>
<p>A <em>linear functional</em> is a map from
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>F</mi><mi>n</mi></msup><mo>→</mo><mi>F</mi></mrow><annotation encoding="application/x-tex">F^n\to F</annotation></semantics></math>.
For all linear functionals
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>Ï•</mi><annotation encoding="application/x-tex">\phi</annotation></semantics></math>,
there is a unique vector
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>v</mi><annotation encoding="application/x-tex">v</annotation></semantics></math>
such that
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>ϕ</mi><mo stretchy="false" form="prefix">(</mo><mi>u</mi><mo stretchy="false" form="postfix">)</mo><mo>=</mo><mo stretchy="false" form="prefix">⟨</mo><mi>u</mi><mo>,</mo><mi>v</mi><mo stretchy="false" form="postfix">⟩</mo></mrow><annotation encoding="application/x-tex">\phi(u) = \langle u, v\rangle</annotation></semantics></math>
for all
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>u</mi><mo>∈</mo><mi>V</mi></mrow><annotation encoding="application/x-tex">u\in V</annotation></semantics></math>.</p>
<p>Fix some vector
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>w</mi><mo>∈</mo><mi>V</mi></mrow><annotation encoding="application/x-tex">w\in V</annotation></semantics></math>.
For a linear map
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>T</mi><mo>:</mo><mi>V</mi><mo>→</mo><mi>W</mi></mrow><annotation encoding="application/x-tex">T\colon V\to W</annotation></semantics></math>,
observe
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false" form="prefix">⟨</mo><mi>T</mi><mi>v</mi><mo>,</mo><mi>w</mi><mo stretchy="false" form="postfix">⟩</mo></mrow><annotation encoding="application/x-tex">\langle Tv, w\rangle</annotation></semantics></math>
is a linear functional from
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>F</mi><mi>n</mi></msup><mo>→</mo><mi>F</mi></mrow><annotation encoding="application/x-tex">F^n\to F</annotation></semantics></math>,
so there exists some
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>T</mi><mo>⋆</mo></msup><mi>w</mi><mo>∈</mo><mi>V</mi></mrow><annotation encoding="application/x-tex">T^{\star} w\in V</annotation></semantics></math>
such that
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false" form="prefix">⟨</mo><mi>T</mi><mi>v</mi><mo>,</mo><mi>w</mi><mo stretchy="false" form="postfix">⟩</mo><mo>=</mo><mo stretchy="false" form="prefix">⟨</mo><mi>v</mi><mo>,</mo><msup><mi>T</mi><mo>⋆</mo></msup><mi>w</mi><mo stretchy="false" form="postfix">⟩</mo></mrow><annotation encoding="application/x-tex">\langle Tv, w\rangle = \langle v, T^{\star}w\rangle</annotation></semantics></math>.
Each
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>w</mi><annotation encoding="application/x-tex">w</annotation></semantics></math>
is associated with a unique
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>T</mi><mo>⋆</mo></msup><mi>w</mi></mrow><annotation encoding="application/x-tex">T^{\star}w</annotation></semantics></math>,
so we can consider
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>T</mi><mo>⋆</mo></msup><annotation encoding="application/x-tex">T^{\star}</annotation></semantics></math>
a function from
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>W</mi><mo>→</mo><mi>V</mi></mrow><annotation encoding="application/x-tex">W\to V</annotation></semantics></math>,
and furthermore,
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>T</mi><mo>⋆</mo></msup><annotation encoding="application/x-tex">T^{\star}</annotation></semantics></math>
is a linear map.</p>
<p>Say now that
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>T</mi><annotation encoding="application/x-tex">T</annotation></semantics></math>
is a linear endomorphism,
i.e. <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>T</mi><mo>:</mo><mi>V</mi><mo>→</mo><mi>V</mi></mrow><annotation encoding="application/x-tex">T\colon V\to V</annotation></semantics></math>.
Then we say
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>T</mi><annotation encoding="application/x-tex">T</annotation></semantics></math>
is <em>self-adjoint</em> if
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>T</mi><mo>=</mo><msup><mi>T</mi><mo>⋆</mo></msup></mrow><annotation encoding="application/x-tex">T=T^{\star}</annotation></semantics></math>
and
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>T</mi><annotation encoding="application/x-tex">T</annotation></semantics></math>
is <em>normal</em> if
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>T</mi><msup><mi>T</mi><mo>⋆</mo></msup><mo>=</mo><msup><mi>T</mi><mo>⋆</mo></msup><mi>T</mi></mrow><annotation encoding="application/x-tex">TT^{\star}=T^{\star}T</annotation></semantics></math>.
Obviously all self-adjoint operators are normal.</p>
<p>The Spectral Theorem states that
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>T</mi><annotation encoding="application/x-tex">T</annotation></semantics></math>
has a diagonal matrix iff</p>
<ul>
<li>In a complex vector space:
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>T</mi><annotation encoding="application/x-tex">T</annotation></semantics></math>
is normal</li>
<li>In a real vector space:
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>T</mi><annotation encoding="application/x-tex">T</annotation></semantics></math>
is self-adjoint</li>
</ul>
<p>I’ve been trying really hard to avoid matrices, but bringing them up
is very natural here. To use terms we have defined before, “has a
diagonal matrix†is equivalent to “has an orthonormal basis of
eigenvectorsâ€.</p>
<h2 id="conclusion">Conclusion</h2>
<p>So that’s linear algebra! A lot of detail has been left out, and we
haven’t even <em>touched</em> matrices, but it’s still quite doable
right?</p>
</body>
</html>