Optimization

Optimization helps us to master the training. It tells us why those simple method (e.g., SGD) can find good parameters.

• Convex Optimization
  • Nesterov, Yurii. Introductory lectures on convex optimization: A basic course. Vol. 87. Springer Science & Business Media, 2013.
  • Boyd, Stephen, Stephen P. Boyd, and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
• Nonconvex Optimization
  • https://sunju.org/research/nonconvex/

Statistic

Statistic characterizes the generalization property of the learning scheme.

• High-dimensional Probability
  • Vershynin, Roman. High-dimensional probability: An introduction with applications in data science. Vol. 47. Cambridge university press, 2018.
  • Wainwright, Martin J. High-dimensional statistics: A non-asymptotic viewpoint. Vol. 48. Cambridge University Press, 2019.

Geometry and Algebra

Geometry and Algebra provide a fundamental tool for understanding a machine learning process.

• Linear Algebra
  • Axler, Sheldon.Linear algebra done right. springer, 2015.
• Advanced
  • Lang, Serge. Fundamentals of differential geometry. Vol. 191. Springer Science & Business Media, 2012.