Bias/Variance

Bias vs Variance

Variance and bias are important concepts of machine learning.

Machine learning models are not perfect, and as such, make errors. Errors are the difference between the predicted value of the model and the actual value. Two of the main errors machine learning tackles are bias and variance. Note: minimising one of the two errors would cause the other to increase – there is always a trade-off between the two.

Bias incurs from false assumptions in the algorithm. It is measured by the difference between the prediction and the actual value. High bias causes higher error in both the training and testing data, which in turn leads to underfitting.

How to mitigate bias:

1. Regularisation reduction
2. Feature engineering
3. Increase model size
4. Change of model architecture

Variance incurs due to the sensitivity of the training data’s changes. It is measured by the degree of how spread the prediction values are. High variance causes higher error in the testing data, which in turn leads to overfitting.

How to mitigate variance:

1. More data
2. Feature selection
3. Decrease regularisation
4. Decrease model size

The following is an example of the trade-off between bias and variance.

Generally, the combination of high bias and low variance associates with underfitting. Conversely, low bias and high variance is a good indication of overfitting.

Next: Supervised Machine Learning