Bias-Variance Tradeoff is a term that many aspiring Data Scientists or Machine Learning Engineers find difficult to understand. The purpose of this article is to try as much as possible to illustrate it in a simple to understand context and to proffer solutions to this dilemma.
As the name connotes Bias-Variance Tradeoff characterizes a critical decision zone a person get to when using predictive machine learning algorithms.it is a point of a tradeoff between two critical choices namely Bias and Variance.
Bias occurs when a machine learning algorithm has limited flexibility to learn the true pattern in a dataset. This also lead to underfitting problem.
Variance occurs when a machine learning algorithm become overly sensitive to or mastering the training data used in training it which makes generalization or use of the machine learning algorithm beyond that particular training data not feasible (overfitting problem)
The role of a machine learning Engineer is to minimize these two problems namely the bias and the variance errors.
The dilemma then lies in the fact that maintaining low Bias relaxes limited flexibility to enable an algorithm to freely learn the true pattern in a dataset however this would be done at the expense of a High Variance. High variance refers to the possibility of the machine learning algorithm to overfit the training data by modeling the random noise in the dataset rather than the intended outputs). Maintaining low bias also, in the long run, will give rise to a too complex model due to the underlying much flexibility and too complex models tend to pick up noise in the dataset rather than the true signal. Regularization can be used to penalize overly complex algorithms in this trade-off. For example, decision trees can be pruned to reduce complexity.
The other side of this dilemma is to maintain a high bias. This would be done at the expense of low variance. High bias ensures that a machine learning algorithm does not overfit the training data by maintaining limited flexibility however this could degenerate into a model having low variance. Low variance means an algorithm is not complex enough to learn the true pattern in training data which makes it underfit the training data. Hence it fails to capture the underlying relationship between features and target output in the dataset. Introducing complexity into the model can boost the underfitting algorithm in this trade-off.
The figure above shows the prediction error as a function of model complexity. As we move to the left, we have high bias but low variance (underfit the data). As we move to the right where there is a higher complexity, we get a lower bias but high variance (overfit the data).
Fixing this Dilemma
Minimizing these two errors (the bias and the variance errors) is the dilemma that needs to be solved, and solving this is much about dealing with over-fitting and under-fitting. The following techniques can be used to address the bias-variance tradeoff:
- Minimizing the Total Error: The goal here is to find the optimum level of complexity at which the increase in bias is equivalent to the reduction in variance. Using a measure of error one can try different levels of model complexity and choose the one that reduces the overall error.
- Use Bagging (Ensemble Algorithm) and Re-Sampling Methods: The aim of bagging is to reduce variance and helps avoid overfitting. Bagging works by creating several subsets of data from the training data chosen randomly with replacement to train several models and then averaging their variance to get a final model whose variance has been greatly reduced. An example of a machine learning algorithm that uses bagging is the Random Forest.
- Splitting your dataset: Splitting your dataset into training data, validation data and test data. The training data is used to train your model, the validation data is used to evaluate the performance of your model if its performance depends on frequent tunning of the hyperparameters, while the testing data is used to evaluate the final performance of your trained model. This process helps to get an optimum model with a less bias-variance error.
- Optimizing your Model by performing Cross-Validation: The dataset is divided into k folds. Each k-1 folds is used to train the algorithm iteratively, while the remaining kth fold (hold-out) is used as the test data. This would be repeated until all the k-folds has been featured as test data. The k recorded errors can then be averaged. This can be used to evaluate the performance of your model and to identify which parameters will result in the lowest test errors.
- Other methods that can be used to solve the problem of bias-variance are: Increasing the size or number of parameters in the model and increase the complexity or type of the model.
