Stanford CS230 | Autumn 2025 | Lecture 9: Career Advice in AI

This lecture teaches you how to choose the best machine learning model and how to evaluate it fairly so it performs well on new, unseen data. The core idea is generalization: we don’t only want a model that is good at memorizing examples it has already seen; we want one that can handle fresh cases tomorrow. You learn the twin pitfalls to avoid: underfitting (the model is too simple and misses patterns) and overfitting (the model is too complex and chases noise). To make selection and evaluation trustworthy, the lecture walks through splitting your dataset into training, validation, and test sets, and explains how cross-validation helps when you have limited data.

The scope covers both classification and regression evaluation metrics, since different problems need different scorecards. For classification, you get accuracy, precision, recall, F1 score, and AUC (area under the ROC curve). For regression, you learn mean squared error (MSE), root mean squared error (RMSE), and R-squared (R²). The lecture also addresses hyperparameter tuning—the process of picking the best settings that are chosen before training—covering grid search, random search, and Bayesian optimization at a conceptual level.

This material is well-suited for beginners who have basic machine learning familiarity: you should know what a dataset is, what a model does, and the meaning of training. If you’ve seen simple models like linear regression or logistic regression, you’ll be comfortable. No advanced math is required beyond understanding that we compare predictions against real labels. If you have tried training a model once and wondered how to judge it properly, this lecture gives you a full, practical framework.

After this lecture, you will be able to: (1) clearly explain underfitting and overfitting, and recognize them in plots or results; (2) set up a clean train/validation/test workflow; (3) run k-fold cross-validation when data is small and average results properly; (4) pick and interpret the right metrics for your problem; and (5) tune hyperparameters using grid search, random search, or Bayesian ideas. You’ll also have the judgment to avoid common traps like peeking at the test set or over-trusting accuracy on imbalanced data, and you’ll know when to adjust thresholds to meet real-world needs, such as catching more true positives in a medical test.

The lecture is structured from first principles to application. It starts with the goal of machine learning—learning a function f(x) that predicts y for new x—and explains why testing only on training data is misleading. It then introduces underfitting and overfitting, with a simple plot showing a linear underfit, a balanced good fit, and an overfit that wiggles through every point. From there, it moves to model selection via the three-way data split, including a typical 70%/15%/15% ratio and when to adjust it. Next, cross-validation is introduced, including common k values like 5 and 10, plus leave-one-out for extreme cases.

With the data-splitting foundation set, the lecture explores evaluation metrics in detail. It starts with accuracy but warns that it can be misleading under class imbalance (e.g., 90% majority class). It then explains precision, recall, their trade-offs, and the F1 score as a harmonic mean. The ROC curve and AUC show how a model ranks positives vs. negatives across thresholds. Finally, the lecture shifts to regression metrics: MSE, RMSE, and R², explaining what each tells you and when they are useful.

The lecture closes with hyperparameter tuning strategies. Grid search tries all combinations in a preset grid (expensive but thorough), random search samples combinations (often faster and effective), and Bayesian optimization uses a probabilistic model to choose the next best set of hyperparameters to test (more sample-efficient). In Q&A, the instructor notes that which hyperparameters matter depends on the model; check documentation and research, and combine knowledge with experimentation. For tiny datasets without a separate validation set, cross-validation is the recommended path. The final message: use solid splits, the right metrics, and thoughtful tuning to pick a model that truly generalizes.

1. Overall Architecture/Structure

Goal and Data Flow

• Input: A dataset of examples, each with features x and a target y (labels for classification or numeric values for regression).
• Training: Fit candidate models on the training set to learn parameters that minimize a loss (e.g., cross-entropy for classification, squared error for regression).
• Validation: Use a validation set, or cross-validation, to evaluate different models and hyperparameters without touching the test set.
• Selection: Choose the best-performing model and hyperparameter setting based on the validation process and chosen metric(s).
• Final Evaluation: Keep the test set strictly separate and run it once at the end to report an unbiased estimate of generalization performance.

Key Roles of Splits

• Training set: Teaches the model by adjusting internal parameters (weights) to reduce error.
• Validation set: Guides decisions among models and settings, giving a less biased view than training error.
• Test set: Acts as the final judge; it must not influence training or tuning decisions to prevent optimistic bias (data leakage).

Bias-Variance Framing

• Underfitting (high bias): The model’s assumptions are too restrictive; it fails to capture patterns, leading to high errors on both training and test data.
• Overfitting (high variance): The model is too sensitive to the specifics of the training set; it learns noise, achieving low training error but high test error.
• Target: Choose model complexity and regularization so that expected test error is minimized.

1. Code/Implementation Details (Conceptual, with examples)

Data Splits

• Standard split: 70% train, 15% validation, 15% test. Adjust as needed based on dataset size.
• Practical tip: Use stratification for classification to keep class ratios similar across splits.
• Random seeds: Fix seeds (e.g., 42) so splits are reproducible.

Cross-Validation

• K-fold CV: Shuffle and partition data into k roughly equal folds (commonly k=5 or k=10). For each fold i from 1 to k, train on folds not equal to i and validate on fold i. Average the metric across k folds to estimate performance.
• Leave-One-Out CV (LOOCV): For each data point, train on all others and validate on that one point. Provides almost full-data training but is computationally heavy and can have high variance per fold.
• When to use: Prefer k-fold CV for small-to-medium datasets; use a single validation split for very large datasets where variance is naturally low.

Choosing Metrics

• Classification: • Accuracy: Fraction correct; simple but can mislead with imbalance. • Precision: TP / (TP + FP); focuses on correctness of positive predictions. • Recall: TP / (TP + FN); focuses on completeness of catching positives. • F1 Score: Harmonic mean of precision/recall; balances the two, especially valuable under imbalance. • ROC and AUC: Plot TPR vs. FPR over thresholds; AUC summarizes ranking ability.
• Regression: • MSE: Average squared error; penalizes large errors strongly. • RMSE: Square root of MSE; interpretable in target units. • R²: Proportion of variance explained; intuitive but be cautious if used alone.

Thresholds and Trade-offs

• Many classifiers output scores or probabilities. Picking a threshold (e.g., 0.5) turns scores into class labels.
• Raising the threshold often increases precision (fewer false positives) but lowers recall (more false negatives), and vice versa.
• Use precision-recall curves and ROC curves to choose thresholds that best match your application’s needs.

Hyperparameters

• Definition: Model settings fixed before training (e.g., learning rate, regularization strength, number of layers, tree depth, kernel choice).
• Examples: • Linear models: Regularization strength (L2 penalty λ), choice of penalty (L1 vs. L2). • Trees/Forests: Max depth, min samples per split, number of trees, feature subsampling. • SVM: C (regularization), kernel type (linear, RBF), kernel parameters (gamma for RBF). • Neural nets: Learning rate, batch size, number of layers/units, dropout rate.

Hyperparameter Tuning Methods

• Grid Search: • Process: Define a discrete grid of values for each hyperparameter and evaluate all combinations. • Pros: Systematic and thorough within the grid; easy to parallelize. • Cons: Scales poorly with more parameters; wastes time on unimportant dimensions. • When to use: Small search spaces, when you want guaranteed coverage.
• Random Search: • Process: Define distributions/ranges and sample random combinations for trial. • Pros: More efficient in high-dimensional spaces; often finds strong settings quickly. • Cons: Results depend on random samples; may miss narrow, optimal regions. • When to use: Larger search spaces; early-stage exploration.
• Bayesian Optimization: • Process: Build a surrogate model (e.g., Gaussian process) of the score as a function of hyperparameters; choose next trials by balancing exploration and exploitation (e.g., via expected improvement). • Pros: Can reach strong results with fewer evaluations; adapts as it learns. • Cons: More complex to implement; overhead may not pay off for tiny problems. • When to use: Expensive training runs where each evaluation costs a lot.

Avoiding Data Leakage

• Never tune hyperparameters on the test set.
• If you try many models and report the best test score, you are implicitly tuning on test; use a separate final holdout or nested cross-validation.
• Carefully separate preprocessing that learns from data (e.g., scaling mean/std) so it is fit only on the training portion within each fold.

Workflow Example (Classification)

1. Prepare data: Clean, handle missing values, encode categorical variables (fit encoders on train only), and stratify splits.
2. Split data:
   • Option A: Train/validation/test (e.g., 70/15/15). Keep test locked.
   • Option B: Train/test plus k-fold CV inside training for tuning.
3. Choose baseline models: For instance, logistic regression and a small decision tree.
4. Define metrics: If classes are balanced and costs symmetric, use accuracy; otherwise, consider F1 and AUC.
5. Tune hyperparameters:
   • For logistic regression: Try C values (inverse of regularization) like {0.01, 0.1, 1, 10}.
   • For decision tree: Try max_depth in {3, 5, 7, 9} and min_samples_split in {2, 10, 20}.
6. Evaluate via validation or cross-validation: Record mean and standard deviation of metrics.
7. Select the best candidate: Consider both average performance and stability across folds.
8. Finalize model: Retrain the chosen configuration on the full training+validation data if using a hold-out validation scheme.
9. Test once: Run on the held-out test set for the final, unbiased estimate.
10. Report results: Include primary metric and supporting metrics (e.g., accuracy, precision, recall, F1, AUC), and document the split protocol.

Workflow Example (Regression)

1. Prepare features: Scale numeric features if needed; be careful to fit scalers on training data only.
2. Split into train/validation/test or set up k-fold CV.
3. Choose models: Linear regression with regularization (Ridge/Lasso) and a random forest regressor.
4. Metrics: Use RMSE for interpretability; report R² to show variance explained.
5. Tuning:
   • Ridge/Lasso: Regularization strength α across a log-scale grid.
   • Random forest: Number of trees, max depth, min samples per leaf.
6. Validate: Use CV to average RMSE across folds.
7. Select and finalize: Pick the model with best RMSE and stable fold performance, retrain on combined training data, then test once.

Evaluation Metric Nuances

• Accuracy Pitfall: In imbalanced datasets (e.g., 99% negatives, 1% positives), predicting all negatives gives 99% accuracy but zero recall on positives. Prefer precision/recall/F1 and AUC.
• Precision vs. Recall: Choose based on costs. In medical screening, missing a sick patient (false negative) may be worse than alarming a healthy person (false positive), so target higher recall. In spam filtering, too many false positives can hide important emails, so precision matters.
• F1 in Practice: Good when you need one number to summarize balance under imbalance. If you value recall more, consider Fβ scores (β>1 weights recall higher); if precision more, use β<1.
• ROC/AUC: Useful when class distributions shift or thresholds vary by use case. For highly imbalanced data, also inspect precision-recall curves because ROC can look optimistic when negatives dominate.
• Regression Choices: MSE punishes large misses strongly; RMSE is interpretable in target units. R² complements them but doesn’t convey error magnitude.

Cross-Validation Details

• Fold construction: Ensure folds are representative; for classification, stratify folds to maintain class ratios.
• Variance estimation: Report mean ± standard deviation across folds to show stability.
• Model refit: After picking the best hyperparameters via CV, refit on full training data (all folds combined) before testing.
• Computational cost: k-fold multiplies training time by k. Balance k with available compute; k=5 or 10 is common.

Hyperparameter Prioritization

• Which to tune: Focus on a few high-impact hyperparameters (e.g., learning rate for neural nets, C and gamma for SVM with RBF, depth for trees). Check documentation and papers to identify these.
• Search ranges: Use log scales for parameters that span orders of magnitude (e.g., 1e-4 to 1e0). Start broad, then narrow.
• Budgeting: Decide on a maximum number of trials or total compute time. Early stopping can save time for models that support it.

Documentation and Reproducibility

• Record: Data split ratios, random seeds, versions of libraries, metric definitions, and chosen thresholds.
• Store: Validation scores for each hyperparameter setting and the final test result. Keep confusion matrices and curves (ROC, precision-recall) where applicable.
• Prevent leakage: Ensure any operation that learns from data (imputation, scaling, feature selection) is applied only using training data within each fold.

Practical Tips and Warnings

• Don’t peek at the test set: Even checking it repeatedly for curiosity introduces bias. Treat test performance as a one-time report.
• Monitor learning curves: Plot training vs. validation error as you vary training set size or epochs. Diverging curves indicate overfitting; high errors on both suggest underfitting.
• Handle imbalance: Try class weighting, resampling (over/under-sampling), or focal loss (in deep learning) in addition to using better metrics.
• Calibrate probabilities: If you care about probability estimates (not just rankings), consider calibration methods like Platt scaling or isotonic regression on validation data.
• Nested CV (advanced): If you must use all data for tuning and testing without a dedicated test set, use nested CV to avoid bias (inner loop for tuning, outer loop for evaluation).

End-to-End Example Summary

• Suppose you have 10,000 email samples with labels spam/ham. You reserve 1,500 for validation and 1,500 for testing, training on 7,000.
• You compare logistic regression vs. a small gradient-boosted tree model. You define F1 as the primary metric due to class imbalance.
• You run random search over 30 trials for each model’s hyperparameters, using 5-fold CV inside the training set to estimate performance, log all results, and pick the best.
• You retrain the chosen configuration on training+validation data, apply a threshold tuned for desired precision/recall balance, and finally evaluate once on the test set to report F1, precision, recall, and AUC alongside a confusion matrix.

This lecture gives you a complete, practical framework for picking and fairly testing machine learning models so they truly generalize. It begins with the core goal—performing well on new data—and uses the twin ideas of underfitting (too simple, high bias) and overfitting (too complex, high variance) to frame the challenge. You learned to structure your process with training, validation, and test sets, including when to adjust the common 70/15/15 split and how cross-validation lets you reuse limited data to get steady estimates. For classification, you now know why accuracy can mislead under imbalance and how precision, recall, F1, and AUC paint a clearer picture. For regression, MSE, RMSE, and R² give complementary views of error size and explanatory power.

On the tuning side, hyperparameters like learning rate and tree depth are key levers for performance. You saw three mainstream strategies: grid search for exhaustive, small spaces; random search for efficient coverage in bigger ones; and Bayesian optimization for sample-efficient improvement guided by a surrogate model. A unifying message is to keep the test set locked away until the very end to avoid data leakage and inflated scores. Also, match your metric and threshold to the real-world costs of mistakes—false positives and false negatives matter differently in different domains.

To practice, start by training two or three simple baseline models on a dataset you know, using a clear train/validation/test split. Try k-fold cross-validation when you have limited data, define one primary metric and a couple of supporting ones, and log results for each hyperparameter trial. Tune a small number of high-impact hyperparameters first, and visualize ROC or precision-recall curves to choose thresholds. Then retrain the selected model on all non-test data and run a single, final test evaluation.

For next steps, learn more about calibration for probabilities, resampling and class weighting for imbalanced data, and nested cross-validation for fully rigorous comparison. Explore advanced model families and regularization techniques, and consider automated hyperparameter tuning libraries. The core message to remember is this: structure your evaluation carefully, choose metrics that match your goals, and separate training, tuning, and testing. If you do, you’ll build models that don’t just look good on paper—they actually work when it counts.