Author: Jason Brownlee
Regression refers to predictive modeling problems that involve predicting a numeric value.
It is different from classification that involves predicting a class label. Unlike classification, you cannot use classification accuracy to evaluate the predictions made by a regression model.
Instead, you must use error metrics specifically designed for evaluating predictions made on regression problems.
In this tutorial, you will discover how to calculate error metrics for regression predictive modeling projects.
After completing this tutorial, you will know:
- Regression predictive modeling are those problems that involve predicting a numeric value.
- Metrics for regression involve calculating an error score to summarize the predictive skill of a model.
- How to calculate and report mean squared error, root mean squared error, and mean absolute error.
Let’s get started.
Regression Metrics for Machine Learning
Photo by Gael Varoquaux, some rights reserved.
Tutorial Overview
This tutorial is divided into three parts; they are:
- Regression Predictive Modeling
- Evaluating Regression Models
- Metrics for Regression
- Mean Squared Error
- Root Mean Squared Error
- Mean Absolute Error
Regression Predictive Modeling
Predictive modeling is the problem of developing a model using historical data to make a prediction on new data where we do not have the answer.
Predictive modeling can be described as the mathematical problem of approximating a mapping function (f) from input variables (X) to output variables (y). This is called the problem of function approximation.
The job of the modeling algorithm is to find the best mapping function we can given the time and resources available.
For more on approximating functions in applied machine learning, see the post:
Regression predictive modeling is the task of approximating a mapping function (f) from input variables (X) to a continuous output variable (y).
Regression is different from classification, which involves predicting a category or class label.
For more on the difference between classification and regression, see the tutorial:
A continuous output variable is a real-value, such as an integer or floating point value. These are often quantities, such as amounts and sizes.
For example, a house may be predicted to sell for a specific dollar value, perhaps in the range of $100,000 to $200,000.
- A regression problem requires the prediction of a quantity.
- A regression can have real-valued or discrete input variables.
- A problem with multiple input variables is often called a multivariate regression problem.
- A regression problem where input variables are ordered by time is called a time series forecasting problem.
Now that we are familiar with regression predictive modeling, let’s look at how we might evaluate a regression model.
Evaluating Regression Models
A common question by beginners to regression predictive modeling projects is:
How do I calculate accuracy for my regression model?
Accuracy (e.g. classification accuracy) is a measure for classification, not regression.
We cannot calculate accuracy for a regression model.
The skill or performance of a regression model must be reported as an error in those predictions.
This makes sense if you think about it. If you are predicting a numeric value like a height or a dollar amount, you don’t want to know if the model predicted the value exactly (this might be intractably difficult in practice); instead, we want to know how close the predictions were to the expected values.
Error addresses exactly this and summarizes on average how close predictions were to their expected values.
There are three error metrics that are commonly used for evaluating and reporting the performance of a regression model; they are:
- Mean Squared Error (MSE).
- Root Mean Squared Error (RMSE).
- Mean Absolute Error (MAE)
There are many other metrics for regression, although these are the most commonly used. You can see the full list of regression metrics supported by the scikit-learn Python machine learning library here:
In the next section, let’s take a closer look at each in turn.
Metrics for Regression
In this section, we will take a closer look at the popular metrics for regression models and how to calculate them for your predictive modeling project.
Mean Squared Error
Mean Squared Error, or MSE for short, is a popular error metric for regression problems.
It is also an important loss function for algorithms fit or optimized using the least squares framing of a regression problem. Here “least squares” refers to minimizing the mean squared error between predictions and expected values.
The MSE is calculated as the mean or average of the squared differences between predicted and expected target values in a dataset.
- MSE = 1 / N * sum for i to N (y_i – yhat_i)^2
Where y_i is the i’th expected value in the dataset and yhat_i is the i’th predicted value. The difference between these two values is squared, which has the effect of removing the sign, resulting in a positive error value.
The squaring also has the effect of inflating or magnifying large errors. That is, the larger the difference between the predicted and expected values, the larger the resulting squared positive error. This has the effect of “punishing” models more for larger errors when MSE is used as a loss function. It also has the effect of “punishing” models by inflating the average error score when used as a metric.
We can create a plot to get a feeling for how the change in prediction error impacts the squared error.
The example below gives a small contrived dataset of all 1.0 values and predictions that range from perfect (1.0) to wrong (0.0) by 0.1 increments. The squared error between each prediction and expected value is calculated and plotted to show the quadratic increase in squared error.
... # calculate error err = (expected[i] - predicted[i])**2
The complete example is listed below.
# example of increase in mean squared error
from matplotlib import pyplot
from sklearn.metrics import mean_squared_error
# real value
expected = [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
# predicted value
predicted = [1.0, 0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2, 0.1, 0.0]
# calculate errors
errors = list()
for i in range(len(expected)):
# calculate error
err = (expected[i] - predicted[i])**2
# store error
errors.append(err)
# report error
print('>%.1f, %.1f = %.3f' % (expected[i], predicted[i], err))
# plot errors
pyplot.plot(errors)
pyplot.xticks(ticks=[i for i in range(len(errors))], labels=predicted)
pyplot.xlabel('Predicted Value')
pyplot.ylabel('Mean Squared Error')
pyplot.show()
Running the example first reports the expected value, predicted value, and squared error for each case.
We can see that the error rises quickly, faster than linear (a straight line).
>1.0, 1.0 = 0.000 >1.0, 0.9 = 0.010 >1.0, 0.8 = 0.040 >1.0, 0.7 = 0.090 >1.0, 0.6 = 0.160 >1.0, 0.5 = 0.250 >1.0, 0.4 = 0.360 >1.0, 0.3 = 0.490 >1.0, 0.2 = 0.640 >1.0, 0.1 = 0.810 >1.0, 0.0 = 1.000
A line plot is created showing the curved or super-linear increase in the squared error value as the difference between the expected and predicted value is increased.
The curve is not a straight line as we might naively assume for an error metric.
Line Plot of the Increase Square Error With Predictions
The individual error terms are averaged so that we can report the performance of a model with regard to how much error the model makes generally when making predictions, rather than specifically for a given example.
The units of the MSE are squared units.
For example, if your target value represents “dollars,” then the MSE will be “squared dollars.” This can be confusing for stakeholders; therefore, when reporting results, often the root mean squared error is used instead (discussed in the next section).
The mean squared error between your expected and predicted values can be calculated using the mean_squared_error() function from the scikit-learn library.
The function takes a one-dimensional array or list of expected values and predicted values and returns the mean squared error value.
... # calculate errors errors = mean_squared_error(expected, predicted)
The example below gives an example of calculating the mean squared error between a list of contrived expected and predicted values.
# example of calculate the mean squared error from sklearn.metrics import mean_squared_error # real value expected = [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] # predicted value predicted = [1.0, 0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2, 0.1, 0.0] # calculate errors errors = mean_squared_error(expected, predicted) # report error print(errors)
Running the example calculates and prints the mean squared error.
0.35000000000000003
A perfect mean squared error value is 0.0, which means that all predictions matched the expected values exactly.
This is almost never the case, and if it happens, it suggests your predictive modeling problem is trivial.
A good MSE is relative to your specific dataset.
It is a good idea to first establish a baseline MSE for your dataset using a naive predictive model, such as predicting the mean target value from the training dataset. A model that achieves an MSE better than the MSE for the naive model has skill.
Root Mean Squared Error
The Root Mean Squared Error, or RMSE, is an extension of the mean squared error.
Importantly, the square root of the error is calculated, which means that the units of the RMSE are the same as the original units of the target value that is being predicted.
For example, if your target variable has the units “dollars,” then the RMSE error score will also have the unit “dollars” and not “squared dollars” like the MSE.
As such, it may be common to use MSE loss to train a regression predictive model, and to use RMSE to evaluate and report its performance.
The RMSE can be calculated as follows:
- RMSE = sqrt(1 / N * sum for i to N (y_i – yhat_i)^2)
Where y_i is the i’th expected value in the dataset, yhat_i is the i’th predicted value, and sqrt() is the square root function.
We can restate the RMSE in terms of the MSE as:
- RMSE = sqrt(MSE)
Note that the RMSE cannot be calculated as the average of the square root of the mean squared error values. This is a common error made by beginners and is an example of Jensen’s inequality.
You may recall that the square root is the inverse of the square operation. MSE uses the square operation to remove the sign of each error value and to punish large errors. The square root reverses this operation, although it ensures that the result remains positive.
The root mean squared error between your expected and predicted values can be calculated using the mean_squared_error() function from the scikit-learn library.
By default, the function calculates the MSE, but we can configure it to calculate the square root of the MSE by setting the “squared” argument to False.
The function takes a one-dimensional array or list of expected values and predicted values and returns the mean squared error value.
... # calculate errors errors = mean_squared_error(expected, predicted, squared=False)
The example below gives an example of calculating the root mean squared error between a list of contrived expected and predicted values.
# example of calculate the root mean squared error from sklearn.metrics import mean_squared_error # real value expected = [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] # predicted value predicted = [1.0, 0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2, 0.1, 0.0] # calculate errors errors = mean_squared_error(expected, predicted, squared=False) # report error print(errors)
Running the example calculates and prints the root mean squared error.
0.5916079783099616
A perfect RMSE value is 0.0, which means that all predictions matched the expected values exactly.
This is almost never the case, and if it happens, it suggests your predictive modeling problem is trivial.
A good RMSE is relative to your specific dataset.
It is a good idea to first establish a baseline RMSE for your dataset using a naive predictive model, such as predicting the mean target value from the training dataset. A model that achieves an RMSE better than the RMSE for the naive model has skill.
Mean Absolute Error
Mean Absolute Error, or MAE, is a popular metric because, like RMSE, the units of the error score match the units of the target value that is being predicted.
Unlike the RMSE, the changes in RMSE are linear and therefore intuitive.
That is, MSE and RMSE punish larger errors more than smaller errors, inflating or magnifying the mean error score. This is due to the square of the error value. The MAE does not give more or less weight to different types of errors and instead the scores increase linearly with increases in error.
As its name suggests, the MAE score is calculated as the average of the absolute error values. Absolute or abs() is a mathematical function that simply makes a number positive. Therefore, the difference between an expected and predicted value may be positive or negative and is forced to be positive when calculating the MAE.
The MAE can be calculated as follows:
- MAE = 1 / N * sum for i to N abs(y_i – yhat_i)
Where y_i is the i’th expected value in the dataset, yhat_i is the i’th predicted value and abs() is the absolute function.
We can create a plot to get a feeling for how the change in prediction error impacts the MAE.
The example below gives a small contrived dataset of all 1.0 values and predictions that range from perfect (1.0) to wrong (0.0) by 0.1 increments. The absolute error between each prediction and expected value is calculated and plotted to show the linear increase in error.
... # calculate error err = abs((expected[i] - predicted[i]))
The complete example is listed below.
# plot of the increase of mean absolute error with prediction error
from matplotlib import pyplot
from sklearn.metrics import mean_squared_error
# real value
expected = [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
# predicted value
predicted = [1.0, 0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2, 0.1, 0.0]
# calculate errors
errors = list()
for i in range(len(expected)):
# calculate error
err = abs((expected[i] - predicted[i]))
# store error
errors.append(err)
# report error
print('>%.1f, %.1f = %.3f' % (expected[i], predicted[i], err))
# plot errors
pyplot.plot(errors)
pyplot.xticks(ticks=[i for i in range(len(errors))], labels=predicted)
pyplot.xlabel('Predicted Value')
pyplot.ylabel('Mean Absolute Error')
pyplot.show()
Running the example first reports the expected value, predicted value, and absolute error for each case.
We can see that the error rises linearly, which is intuitive and easy to understand.
>1.0, 1.0 = 0.000 >1.0, 0.9 = 0.100 >1.0, 0.8 = 0.200 >1.0, 0.7 = 0.300 >1.0, 0.6 = 0.400 >1.0, 0.5 = 0.500 >1.0, 0.4 = 0.600 >1.0, 0.3 = 0.700 >1.0, 0.2 = 0.800 >1.0, 0.1 = 0.900 >1.0, 0.0 = 1.000
A line plot is created showing the straight line or linear increase in the absolute error value as the difference between the expected and predicted value is increased.
Line Plot of the Increase Absolute Error With Predictions
The mean absolute error between your expected and predicted values can be calculated using the mean_absolute_error() function from the scikit-learn library.
The function takes a one-dimensional array or list of expected values and predicted values and returns the mean absolute error value.
... # calculate errors errors = mean_absolute_error(expected, predicted)
The example below gives an example of calculating the mean absolute error between a list of contrived expected and predicted values.
# example of calculate the mean absolute error from sklearn.metrics import mean_absolute_error # real value expected = [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] # predicted value predicted = [1.0, 0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2, 0.1, 0.0] # calculate errors errors = mean_absolute_error(expected, predicted) # report error print(errors)
Running the example calculates and prints the mean absolute error.
0.5
A perfect mean absolute error value is 0.0, which means that all predictions matched the expected values exactly.
This is almost never the case, and if it happens, it suggests your predictive modeling problem is trivial.
A good MAE is relative to your specific dataset.
It is a good idea to first establish a baseline MAE for your dataset using a naive predictive model, such as predicting the mean target value from the training dataset. A model that achieves a MAE better than the MAE for the naive model has skill.
Further Reading
This section provides more resources on the topic if you are looking to go deeper.
Tutorials
- How Machine Learning Algorithms Work
- Difference Between Classification and Regression in Machine Learning
APIs
- Scikit-Learn API: Regression Metrics.
- Scikit-Learn User Guide Section 3.3.4. Regression metrics.
- sklearn.metrics.mean_squared_error API.
- mean_absolute_error API.
Articles
- Mean squared error, Wikipedia.
- Root-mean-square deviation, Wikipedia.
- Mean absolute error, Wikipedia.
- Coefficient of determination, Wikipedia.
Summary
In this tutorial, you discovered how to calculate error for regression predictive modeling projects.
Specifically, you learned:
- Regression predictive modeling are those problems that involve predicting a numeric value.
- Metrics for regression involve calculating an error score to summarize the predictive skill of a model.
- How to calculate and report mean squared error, root mean squared error, and mean absolute error.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
The post Regression Metrics for Machine Learning appeared first on Machine Learning Mastery.