TTIC 101(CMSC 35420): Statistical Methods for Artificial Intelligence, Autumn 2006

Instructors: Sham Kakade and David McAllester

MWF 1:35-2:20 TTIC 230 (Press Buidling)

Course Description

Grading: The course will have roughly one homework set per week, one midterm, and one final.

Description: This course gives a survey of mathematical methods in statistical modeling, inference, and learning with an emphasis on techniques widely used in speech recognition, computational linguistics, and vision. This course is aimed at providing students with a core understanding of statistical AI.

The course will be roughly divided into three Parts, namely models, supervised learning, and unsupervised learning. A tentative outline is as follows.

Spring 2005 course page

Problem Sets

General Outline (Subject to Change)

Part I: Information Theory, Modeling, Perception and Inference.

·       Entropy and Data Compression, Spring 05 notes Autumn 06 notes

·       Hidden Markov Models. Viterbi and Forward-Backward.

·       Probabilistic Context Free Grammars (PCFGs).  Viterbi and Inside-Outside.

·       Linear Dynamical Systems and the Kalman Filter.

·       Viterbi vs. A*

·       Bayesian Networks, Markov Random Fields, and Recursive Conditioning

·       Junction Trees, Tree Width and the Running Time of Recursive Conditioning

·       Loopy Belief Propagation

Part II: Supervised Learning

·       Occam's Razor Theorem

·       Least Squares and the Bias Variance Tradeoff

·       Linear Regression

·       Linear Regression in the Orthogonal Case

·       L1 and L2 Regularization

·       Linear Discriminant Analysis

·       Regularized Regression with Square Loss, Logistic Loss, Hinge Loss, and Sigmoidal Loss

·       The Representor Theorem, Kernels, and Hilbert Spaces.

·       Decision Trees

·       Boosting

·       Structured Labels

·       Feature Selection

Part III: Unsupervised Learning

·       K Means

·       Expectation Maximization (EM)

One More

·       Graph Cuts for MRF Inference