Stanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 6 - LLM Reasoning

This lecture introduces the foundations of machine learning (ML) in a friendly, practical way. It starts by defining ML using Arthur Samuel’s classic idea: computers learn from data without being explicitly programmed. The lecture contrasts hand-crafted rules—like trying to define “cat vs. dog” with ear shapes and nose sizes—with the ML approach of feeding labeled examples and letting the computer discover patterns on its own. From there, it explains why ML is valuable: it tackles complex problems where rules are hard to write, adapts to changing data, and automates tasks such as spam filtering or parts of self-driving. The lecture then maps the ML landscape into three main types: supervised learning, unsupervised learning, and reinforcement learning. Supervised learning uses labeled data to learn functions that map inputs (features) to outputs (labels). Two key branches are covered: classification (predicting categories) and regression (predicting numbers). For regression, the lecture presents linear regression, explains least squares, and shows the closed-form formulas for slope (m) and intercept (b). For binary classification, it introduces logistic regression, the sigmoid function to output probabilities, and the need for iterative optimization like gradient descent to fit weights. Next, the lecture explores unsupervised learning, which discovers structure without labels. It explains clustering, focusing on k-means: initialize centroids, assign points to the nearest centroid, recompute centroids, and repeat until stable. It then covers dimensionality reduction via principal component analysis (PCA): center data, compute the covariance matrix, find eigenvectors/eigenvalues, sort by variance explained, choose top k components, and project data onto them. These methods help with segmentation, anomaly detection, visualization, and pre-processing. Reinforcement learning (RL) is introduced as learning by interaction and reward. An agent observes a state, takes an action, receives a reward, and transitions to a new state. The goal is to learn a policy that maximizes long-term reward. The lecture highlights Q-learning, which maintains a table of expected rewards for state-action pairs and updates it using a simple, powerful rule that combines immediate reward and the best estimated future return. The lecture then tackles key challenges: overfitting (memorizing training data and failing on new data), the bias-variance tradeoff (too simple vs. too complex), and interpretability (understanding model decisions). It offers practical solutions: regularization penalties to control complexity, cross-validation to estimate generalization, and feature importance or simpler models for transparency. These habits keep models reliable and trustworthy. Finally, the lecture lists essential tools and practical next steps. Python and R are popular languages; scikit-learn provides classic ML algorithms; TensorFlow and PyTorch support deep learning. It clarifies that ML is a subset of AI and outlines useful math foundations: linear algebra, calculus, probability, and statistics. To continue learning, it recommends structured online courses and diving into documentation like scikit-learn’s, using small projects to turn theory into skill. By the end, you understand the ML toolbox, when to use each part, and how to keep models robust and interpretable.

Overall Architecture/Structure

• Supervised Learning Workflow
  1. Problem framing: Decide if the task predicts categories (classification) or numbers (regression). Clarify inputs (features) and outputs (labels).
  2. Data collection and cleaning: Gather labeled examples; remove obvious errors; handle missing values.
  3. Feature preparation: Scale/normalize numeric features; one-hot encode categorical features; split into train/validation/test sets.
  4. Model selection: Start simple (linear/logistic regression) as baselines; consider more complex models later if needed.
  5. Training: Optimize parameters by minimizing a loss function (e.g., mean squared error for regression; log loss for classification).
  6. Validation and tuning: Use cross-validation to pick hyperparameters (like regularization strength). Monitor overfitting.
  7. Evaluation: Use appropriate metrics (R^2/MSE for regression; accuracy/precision/recall/AUC for classification).
  8. Deployment: Package the model with preprocessing steps; monitor performance drift; retrain as data evolves.
• Unsupervised Learning Workflow
  1. Problem framing: Decide if you want clusters (grouping) or reduced dimensions (compression/visualization).
  2. Data preparation: Standardize features (especially for PCA and k-means); remove gross outliers or cap them.
  3. Algorithm run: For k-means, pick k and iterate assignments/centroids; for PCA, compute components.
  4. Evaluation: For k-means, inspect cluster separation and usefulness; for PCA, check explained variance ratios.
  5. Use results: Label clusters with business-friendly names; feed reduced features into later models.
• Reinforcement Learning Workflow
  1. Define environment: States, actions, rewards, and transitions.
  2. Agent design: Choose a table-based learner (Q-learning) or function approximation (for large spaces).
  3. Exploration strategy: ε-greedy (with probability ε choose a random action) to balance trying new moves and using known good ones.
  4. Learning loop: Interact over episodes, update value estimates (Q-values), and evaluate cumulative reward.
  5. Convergence: Decay ε and learning rate α; ensure sufficient exploration for stable policies.

Code/Implementation Details (illustrative; Python/scikit-learn style)

• Linear Regression (Closed Form) Idea: Fit y = mx + b by minimizing sum of squared residuals. In vector/matrix form with multiple features X (n×d) and target y (n×1), the closed-form solution is w = (X^T X)^{-1} X^T y (with optional intercept handled by adding a column of ones). For one feature, slope m = Σ(xi−x̄)(yi−ȳ) / Σ(xi−x̄)^2, intercept b = ȳ − m x̄. Steps:
  1. Add a bias column of ones to X to learn intercept b.
  2. Compute XtX = X^T X and Xty = X^T y.
  3. Solve XtX w = Xty (use a stable solver like np.linalg.lstsq instead of explicit inverse).
  4. Predict with ŷ = X w. scikit-learn: from sklearn.linear_model import LinearRegression; model.fit(X_train, y_train); y_pred = model.predict(X_test).
• Linear Regression (Gradient Descent)
  1. Initialize weights w randomly; choose learning rate η.
  2. For each epoch, compute predictions ŷ = X w.
  3. Compute gradient ∇ = (2/n) X^T (ŷ − y).
  4. Update w ← w − η ∇; repeat until convergence. Tip: Standardize features to improve stability and speed.
• Logistic Regression (Binary) Model: p = σ(w^T x + b), where σ(z) = 1/(1+e^{−z}). Loss: Log loss = −[y log p + (1−y) log (1−p)] averaged over samples. No closed-form solution; use gradient descent or quasi-Newton (e.g., LBFGS). Gradient:
  1. Compute p for each sample.
  2. Gradient of loss w.r.t w is X^T (p − y) / n (plus regularization term if used).
  3. Update w ← w − η ∇. Use learning rate schedules or solvers with line search. scikit-learn: from sklearn.linear_model import LogisticRegression; LogisticRegression(penalty='l2', solver='lbfgs').fit(X_train, y_train).
• Regularization (L2 and L1) L2 (ridge): Add λ ||w||^2 to loss; shrinks weights smoothly, reduces variance. L1 (lasso): Add λ ||w||_1; can set some weights to zero, aiding feature selection. In logistic regression, both help prevent overfitting; in linear regression, ridge has closed-form w = (X^T X + λI)^{-1} X^T y. scikit-learn: Ridge, Lasso, ElasticNet (mix of L1 and L2).
• K-Means Clustering
  1. Choose k and initialize centroids (k-means++ is a smart, spread-out initializer).
  2. Assignment step: assign each point to nearest centroid by Euclidean distance.
  3. Update step: recompute centroids as mean of assigned points.
  4. Repeat 2–3 until centroids change little or max iterations reached. Tips: Scale features; run multiple initializations (n_init) to avoid poor local minima; evaluate with inertia and silhouette scores. scikit-learn: from sklearn.cluster import KMeans; KMeans(n_clusters=k, n_init=10).fit(X).
• PCA (Principal Component Analysis)
  1. Standardize/center features: subtract mean; often scale to unit variance.
  2. Compute covariance matrix Σ = (1/(n−1)) X_centered^T X_centered.
  3. Compute eigenvalues/eigenvectors of Σ; sort by descending eigenvalues.
  4. Select top k eigenvectors (principal components) and form projection matrix W.
  5. Transform data: Z = X_centered W. Tips: Inspect explained_variance_ratio_ to pick k; beware of outliers skewing components. scikit-learn: from sklearn.decomposition import PCA; PCA(n_components=k).fit_transform(X).
• Cross-Validation K-fold CV: Split into k folds; for each fold, train on k−1 folds and test on the remaining fold; average metrics. Use StratifiedKFold for classification to preserve class ratios. For hyperparameter tuning, use GridSearchCV or RandomizedSearchCV with a CV splitter. scikit-learn: from sklearn.model_selection import cross_val_score, StratifiedKFold, GridSearchCV.
• Reinforcement Learning: Q-Learning
  1. Initialize Q(s,a) arbitrarily (often zeros) for all state-action pairs.
  2. For each episode, start from an initial state s.
  3. Choose action a using ε-greedy: with probability ε choose random action, else a = argmax_a Q(s,a).
  4. Take action, observe reward r and new state s'.
  5. Update: Q(s,a) ← Q(s,a) + α [r + γ max_{a'} Q(s',a') − Q(s,a)].
  6. Set s ← s' and repeat until episode ends; decay ε over time. Tips: Choose α (learning rate) small enough for stability; γ controls how far ahead you plan. For large state spaces, move from tables to function approximation (e.g., neural networks, “Deep Q-Networks”).

Tools/Libraries Used

• Python: General-purpose, rich ecosystem (NumPy, pandas, scikit-learn, matplotlib, PyTorch, TensorFlow). Chosen for readability and community support.
• R: Strong in statistics and visualization (ggplot2, caret). Great for exploratory analysis and academic settings.
• scikit-learn: Classic ML algorithms with consistent APIs for preprocessing, models, pipelines, and model selection.
• TensorFlow and PyTorch: Deep learning frameworks with GPU support and automatic differentiation. Choose based on team familiarity and deployment needs.

Step-by-Step Implementation Guide (End-to-End Example)

1. Define the task: Predict whether an email is spam (classification) and predict its approximate length in characters (regression) for a bonus metric.
2. Collect data: Emails with labels (spam/not spam) and recorded lengths.
3. Preprocess:
   • Clean text (lowercase, remove obvious noise).
   • Convert text to features (e.g., TF-IDF vectors) and standardize numeric features (length).
   • Split into train/validation/test (e.g., 70/15/15) using StratifiedKFold for classification balance.
4. Baseline models:
   • LogisticRegression for spam classification with L2 penalty; evaluate accuracy, precision, recall, and ROC AUC via 5-fold CV.
   • LinearRegression with TF-IDF dimension reduced by PCA for length prediction; evaluate with RMSE and R^2.
5. Tuning:
   • Use GridSearchCV to try different C values (inverse regularization strength) for logistic regression.
   • Try Ridge for regression and pick α via cross-validation.
6. Final evaluation:
   • Train best hyperparameters on combined train+validation, then test once on the held-out test set.
   • Inspect feature importance (e.g., logistic coefficients) for interpretability.
7. Deployment:
   • Build a scikit-learn Pipeline that includes preprocessing (TF-IDF, PCA) and the model.
   • Serialize with joblib; monitor production metrics and retrain periodically as data drifts.

Tips and Warnings

• Guard against data leakage: Never let test information influence training, including through preprocessing fit steps—always fit transformations on train only.
• Scale features for distance-based methods (k-means) and gradient descent stability.
• Start simple: Linear/logistic regression often perform surprisingly well and are interpretable.
• Use multiple metrics: Accuracy can mislead with imbalanced classes; add precision, recall, and ROC AUC.
• Choose k thoughtfully in k-means: Use the elbow method and silhouette score, but also validate clusters’ business meaning.
• PCA before clustering: Reduces noise and helps k-means on high-dimensional data.
• Monitor models in production: Watch input distributions and performance; set retraining triggers.
• RL pitfalls: Insufficient exploration (ε too small) stalls learning; too large α causes instability; bad reward shaping leads to unintended behaviors.

Illustrative Examples (Inputs → Processing → Outputs)

• Cat vs Dog Images: Input thousands of labeled photos. Train a classifier to map pixel features to labels; evaluate accuracy on new photos. Output: probabilities for “cat” and “dog,” choose the higher. Emphasizes learning patterns over brittle ear/nose rules.
• Heart Attack Risk: Input patient records (age, blood pressure, cholesterol). Train a logistic regression; sigmoid outputs risk probability. Doctors get a calibrated risk score to support decisions. Shows ML handling complex, multi-factor predictions.
• Spam Filtering: Input email text and metadata. Transform text into TF-IDF features and fit logistic regression; update with new labeled emails over time. Output: spam probability and label. Demonstrates automation and adaptation to changing spam tactics.
• House Price Prediction: Input features like size, rooms, location. Fit linear/ridge regression; use cross-validation to tune regularization. Output: predicted price and residual analysis. Highlights regression and overfitting control.
• Ad Click Prediction: Input user/device/time/ad features. Logistic regression estimates click probability; threshold picks the positive class. Output: CTR estimate per impression. Shows sigmoid-based probability modeling.
• Customer Segmentation: Input purchase histories and behavior metrics. Standardize features; run k-means with various k and pick via silhouette score. Output: cluster labels with distinct profiles (e.g., bargain hunters, loyal buyers). Illustrates unsupervised grouping.
• Anomaly Detection: Input transaction features. Fit clustering or density model; flag points far from any cluster. Output: anomaly scores for fraud investigation. Shows unsupervised detection of unusual behavior.
• PCA for Visualization: Input hundreds of numeric features. Center data; compute PCA; project to 2D. Output: a scatterplot revealing natural groupings or gradients. Demonstrates dimensionality reduction.
• Robot Navigation (RL): Input: grid map with start/goal; rewards for reaching goal, small penalties per move. Q-learning updates a table while exploring. Output: a policy that chooses actions to reach the goal efficiently. Shows sequential decision learning.
• Overfitting Demo: Input a small dataset with noise. Fit a very flexible model and a regularized model; compare train vs test errors. Output: regularized model generalizes better. Teaches the danger of memorizing noise.
• Cross-Validation in Practice: Input labeled dataset. Run 5-fold CV for several C values in logistic regression. Output: mean AUC per C and the best setting. Shows robust hyperparameter selection.
• Feature Importance for Trust: Input trained logistic regression for fraud. Inspect top positive and negative coefficients. Output: a ranked list of influential features to guide analysts. Builds interpretability.
• Drift and Adaptation: Input production model facing new data distribution (e.g., seasonality). Monitor drop in metrics; retrain with recent data. Output: restored performance. Shows ML’s ability to adapt over time.

This introduction has built a complete map of machine learning’s core ideas, methods, and practices. You learned that ML lets computers learn patterns from data instead of brittle rules, which makes it powerful for complex, changing problems. We explored the three pillars: supervised learning for labeled prediction, unsupervised learning for structure discovery, and reinforcement learning for decision-making through rewards. Within supervised learning, linear regression (with least squares) and logistic regression (with sigmoid and gradient descent) provide simple, strong baselines for numeric and binary tasks. In unsupervised learning, k-means clustering and PCA reveal groups and compress information, helping both understanding and downstream modeling. Reinforcement learning’s Q-learning showed how agents can learn good actions through iterative updates balancing immediate and future rewards. You also learned the practical guardrails that keep models honest: prevent overfitting with regularization and cross-validation, and use the bias-variance tradeoff to pick the right complexity. Interpretability and feature importance help you trust and debug models, especially in sensitive domains. On the tooling side, Python, R, scikit-learn, TensorFlow, and PyTorch give you everything you need to go from idea to deployment. The difference between ML and AI clarifies language and scope, and the math foundations (linear algebra, calculus, probability, statistics) deepen your intuition and skill. To make this real, start small: build a spam filter with logistic regression, a house price predictor with linear or ridge regression, and a customer segmentation with k-means and PCA. Use cross-validation for tuning, inspect feature importance, and document your pipeline. As a next step, try more advanced models or move into deep learning when your data and problems demand it. The key message to remember is to learn by doing: simple, well-validated models, clear evaluation, and steady iteration will carry you far. With these fundamentals, you can confidently approach new ML problems, choose suitable methods, and deliver reliable, interpretable results.