Table of contents

1. Foundations

• Supervised Learning: input → output mapping using labeled data.
• Key concepts: train/test split, cross-validation, bias–variance tradeoff, overfitting, underfitting.
• Math prerequisites: linear algebra, calculus (gradients), probability & statistics, optimization (GD, SGD, LBFGS).

2. Linear & Regularized Regression

Regression algorithms for continuous targets; many have classification counterparts (e.g., logistic).

• Ordinary Least Squares (OLS) / Simple Linear Regression / Multiple Linear Regression
• Polynomial Regression (basis expansion)
• Ridge Regression (L2)
• Lasso Regression (L1)
• Elastic Net (L1 + L2)
• Multi-task Lasso / Multi-task Elastic Net
• Least Angle Regression (LARS) & LARS-Lasso
• Orthogonal Matching Pursuit (OMP)
• Quantile Regression
• Bayesian Linear Regression (ARD, Bayesian Ridge)
• Robust Regression (Huber, RANSAC)
• Kernel Ridge Regression (ridge with kernel trick)
• Stochastic Gradient Descent (SGD) linear models (for large-scale learning)

3. Margin & Kernel Methods (SVM / SVR)

• Support Vector Machines (SVC) — linear & kernelized (RBF, polynomial, sigmoid)
• Support Vector Regression (SVR)
• Kernel SVM variants & kernelized algorithms (kernel ridge, kernel PCA)

4. Discriminant Analysis

• Linear Discriminant Analysis (LDA)
• Quadratic Discriminant Analysis (QDA)
• Fisher’s Linear Discriminant (classical formulation)

5. Instance-Based Methods

• k-Nearest Neighbors (kNN) — classification & regression
• Weighted kNN / radius neighbors
• Nearest Centroid
• Neighborhood Components Analysis (NCA)

6. Probabilistic & Bayesian Methods

• Naïve Bayes family: Gaussian, Multinomial, Bernoulli, Complement NB
• Bayesian Logistic Regression
• Gaussian Process Regression (GPR)
• Gaussian Process Classification (GPC)
• Hidden Markov Models (supervised sequence classification setting)

7. Decision Trees & Ensemble Methods

Tree learners and ensembles are essential — many modern state-of-the-art solutions are boosted ensembles.

• Decision Trees: ID3, C4.5, CART (classification & regression trees)
• Bagging (Bootstrap Aggregating)
• Random Forests (Breiman)
• Extra Trees (Extremely Randomized Trees)
• AdaBoost (adaptive boosting)
• Gradient Boosting Machines (GBM)
• Advanced boosting frameworks: XGBoost, LightGBM, CatBoost
• Stochastic Gradient Boosting
• Stacking (stacked generalization)
• Voting Classifier / Regressor (hard/soft voting)

8. Decomposition & Dimensionality Techniques (used in supervised pipelines)

• Partial Least Squares (PLS): PLSRegression, PLSCanonical, PLSSVD
• Canonical Correlation Analysis (CCA)
• Linear / Kernel PCA used as preprocessing

9. Online, Streaming & Robust Learners

• Stochastic Gradient Descent (SGD) classifiers & regressors
• Passive Aggressive algorithms (online large-margin classifiers)
• Online versions of Naïve Bayes, Perceptron
• Incremental decision tree learners (e.g., Hoeffding Trees)
• Robust methods for outliers: HuberRegressor, RANSAC

10. Special Topics & Variants

• Neural Networks & Deep Learning (MLP, CNN, RNN, Transformers) for classification & regression
• Multilabel & Multioutput techniques (One-vs-Rest, One-vs-One, Classifier chains, MultiOutputRegressor)
• Imbalanced learning techniques (SMOTE, ADASYN, class weighting, focal loss)
• Calibration methods (Platt scaling, isotonic regression)
• Metric learning (triplet loss, contrastive loss)
• Explainability & interpretability: SHAP, LIME, feature importance, partial dependence plots
• Time-series supervised variants (ARIMA, SARIMAX, supervised feature-based regressors; note: many time-series models are specialized)
• Survival analysis (Cox proportional hazards — regression-like)
• AutoML frameworks (auto-sklearn, TPOT, H2O AutoML, Google AutoML)

Summary Table