Topics in Numerical Analysis

The nicer and practical parts of linear algebra, differential equations, and optimization.

---

Numerical Linear Algebra

Numerical linear algebra is where rubber meets the road. This post collects some topics of interest to me. I aim to provide exposition better than standard textbooks whenever I can. Often times, ideas keep lying around deep inside 1000-page bibles.

• Hutchinson Trace Estimator
• Cholesky decomposition
  • Pivoted Cholesky decomposition
• Preconditioning
• Conjugate gradients
  • Modified Batched Conjugate Gradient Descent
• Lanczos tridiagonalization
• Kronecker-factored matrices
• Toeplitz matrices
  • Szegö's theorem

References

An uncategorized list of references of high pedagogic value.

• An Introduction to the Conjugate Gradient Method Without the Agonizing Pain by Jonathan Richard Shewchuk (1994)

• Toeplitz and Circulant Matrices: A Review by Robert M. Gray (2006)

• Scalable Inference for Structured Gaussian Process Models, Chapter 5 ¹ by Yunus Saatçi (2011)

• The Matrix Cookbook by Kaare Brandt Petersen, Michael Syskind Pedersen (2012)

• Discovering Transforms: A Tutorial on Circulant Matrices, Circular Convolution, and the Discrete Fourier Transform by Bassam Bamieh (2018)

Lectures

• Scientific Computing for DPhil Students by Nick Trefethen (2016)

Textbooks

• Numerical Linear Algebra by Lloyd N. Trefethen and David Bau, III (1997)

• Numerical Optimization by Jorge Nocedal, Stephen J. Wright (2006)

Footnotes

1. Includes description, properties and an application of Kronecker-factored matrices. ↩