How-to-Train-to-the-Test-Set-in-Machine-Learning.jpg

How to Train to the Test Set in Machine Learning

Training to the test set is a type of overfitting where a model is prepared that intentionally achieves good performance on a given test set at the expense of increased generalization error.

It is a type of overfitting that is common in machine learning competitions where a complete training dataset is provided and where only the input portion of a test set is provided. One approach to training to the test set involves constructing a training set that most resembles the test set and then using it as the basis for training a model. The model is expected to have better performance on the test set, but most likely worse performance on the training dataset and on any new data in the future.

Although overfitting the test set is not desirable, it can be interesting to explore as a thought experiment and provide more insight into both machine learning competitions and avoiding overfitting generally.

In this tutorial, you will discover how to intentionally train to the test set for classification and regression problems.

After completing this tutorial, you will know:

  • Training to the test set is a type of data leakage that may occur in machine learning competitions.
  • One approach to training to the test set involves creating a training dataset that is most similar to a provided test set.
  • How to use a KNN model to construct a training dataset and train to the test set with a real dataset.

Kick-start your project with my new book Data Preparation for Machine Learning, including step-by-step tutorials and the Python source code files for all examples.

Let’s get started.

How to Train to the Test Set in Machine Learning

How to Train to the Test Set in Machine Learning
Photo by ND Strupler, some rights reserved.

Tutorial Overview

This tutorial is divided into three parts; they are:

  • Train to the Test Set
  • Train to Test Set for Classification
  • Train to Test Set for Regression
  • Train to the Test Set

    In applied machine learning, we seek a model that learns the relationship between the input and output variables using the training dataset.

    The hope and goal is that we learn a relationship that generalizes to new examples beyond the training dataset. This goal motivates why we use resampling techniques like k-fold cross-validation to estimate the performance of the model when making predictions on data not used during training.

    In the case of machine learning competitions, like those on Kaggle, we are given access to the complete training dataset and the inputs of the test dataset and are required to make predictions for the test dataset.

    This leads to a possible situation where we may accidentally or choose to train a model to the test set. That is, tune the model behavior to achieve the best performance on the test dataset rather than develop a model that performs well in general, using a technique like k-fold cross-validation.

    Another, more overt path to information leakage, can sometimes be seen in machine learning competitions where the training and test set data are given at the same time.

    — Page 56, Feature Engineering and Selection: A Practical Approach for Predictive Models, 2019.

    Training to the test set is often a bad idea.

    It is an explicit type of data leakage. Nevertheless, it is an interesting thought experiment.

    One approach to training to the test set is to contrive a training dataset that is most similar to the test set. For example, we could discard all rows in the training set that are too different from the test set and only train on those rows in the training set that are maximally similar to rows in the test set.

    While the test set data often have the outcome data blinded, it is possible to “train to the test” by only using the training set samples that are most similar to the test set data. This may very well improve the model’s performance scores for this particular test set but might ruin the model for predicting on a broader data set.

    — Page 56, Feature Engineering and Selection: A Practical Approach for Predictive Models, 2019.

    We would expect the model to overfit the test set, but this is the whole point of this thought experiment.

    Let’s explore this approach to training to the test set in this tutorial.

    We can use a k-nearest neighbor model to select those instances of the training set that are most similar to the test set. The KNeighborsRegressor and KNeighborsClassifier both provide the kneighbors() function that will return indexes into the training dataset for rows that are most similar to a given data, such as a test set.


    We might want to try removing duplicates from the selected row indexes.


    We can then use those row indexes to construct a custom training dataset and fit a model.


    Given that we are using a KNN model to construct the training set from the test set, we will also use the same type of model to make predictions on the test set. This is not required, but it makes the examples simpler.

    Using this approach, we can now experiment with training to the test set for both classification and regression datasets.



    Want to Get Started With Data Preparation?

    Take my free 7-day email crash course now (with sample code).

    Click to sign-up and also get a free PDF Ebook version of the course.

    Download Your FREE Mini-Course


    Train to Test Set for Classification

    We will use the diabetes dataset as the basis for exploring training for the test set for classification problems.

    Each record describes the medical details of a female and the prediction is the onset of diabetes within the next five years.

    The dataset has eight input variables and 768 rows of data; the input variables are all numeric and the target has two class labels, e.g. it is a binary classification task.

    Below provides a sample of the first five rows of the dataset.


    First, we can load the dataset directly from the URL, split it into input and output elements, then split the dataset into train and test sets, holding thirty percent back for the test set. We can then evaluate a KNN model with default model hyperparameters by training it on the training set and making predictions on the test set.

    The complete example is listed below.


    Running the example first loads the dataset and summarizes the number of rows and columns, matching our expectations. The shape of the train and test sets are then reported, showing we have about 230 rows in the test set.

    Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

    Finally, the classification accuracy of the model is reported to be about 77.056 percent.


    Now, let’s see if we can achieve better performance on the test set by preparing a model that is trained directly for it.

    First, we will construct a training dataset using the simpler example in the training set for each row in the test set.


    Next, we will train the model on this new dataset and evaluate it on the test set as we did before.


    The complete example is listed below.


    Running the example, we can see that the reported size of the new training dataset is the same size as the test set, as we expected.

    Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

    We can see that we have achieved a lift in performance by training to the test set over training the model on the entire training dataset. In this case, we achieved a classification accuracy of about 79.654 percent compared to 77.056 percent when the entire training dataset is used.


    You might want to try selecting different numbers of neighbors from the training set for each example in the test set to see if you can achieve better performance.

    Also, you might want to try keeping unique row indexes in the training set and see if that makes a difference.

    Finally, it might be interesting to hold back a final validation dataset and compare how different “train-to-the-test-set” techniques affect performance on the holdout dataset. E.g. see how training to the test set impacts generalization error.

    Report your findings in the comments below.

    Now that we know how to train to the test set for classification, let’s look at an example for regression.

    Train to Test Set for Regression

    We will use the housing dataset as the basis for exploring training for the test set for regression problems.

    The housing dataset involves the prediction of a house price in thousands of dollars given details of the house and its neighborhood.

    It is a regression problem, meaning we are predicting a numerical value. There are 506 observations with 13 input variables and one output variable.

    A sample of the first five rows is listed below.


    First, we can load the dataset, split it, and evaluate a KNN model on it directly using the entire training dataset. We will report performance on this regression class using mean absolute error (MAE).

    The complete example is listed below.


    Running the example first loads the dataset and summarizes the number of rows and columns, matching our expectations. The shape of the train and test sets are then reported, showing we have about 150 rows in the test set.

    Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

    Finally, the MAE of the model is reported to be about 4.488.


    Now, let’s see if we can achieve better performance on the test set by preparing a model that is trained to it.

    First, we will construct a training dataset using the simpler example in the training set for each row in the test set.


    Next, we will train the model on this new dataset and evaluate it on the test set as we did before.


    The complete example is listed below.


    Running the example, we can see that the reported size of the new training dataset is the same size as the test set, as we expected.

    Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

    We can see that we have achieved a lift in performance by training to the test set over training the model on the entire training dataset. In this case, we achieved a MAE of about 4.433 compared to 4.488 when the entire training dataset is used.

    Again, you might want to explore using a different number of neighbors when constructing the new training set and see if keeping unique rows in the training dataset makes a difference. Report your findings in the comments below.


    Further Reading

    This section provides more resources on the topic if you are looking to go deeper.

    Books
    APIs

    Summary

    In this tutorial, you discovered how to intentionally train to the test set for classification and regression problems.

    Specifically, you learned:

    • Training to the test set is a type of data leakage that may occur in machine learning competitions.
    • One approach to training to the test set involves creating a training dataset that is most similar to a provided test set.
    • How to use a KNN model to construct a training dataset and train to the test set with a real dataset.

    Do you have any questions?
    Ask your questions in the comments below and I will do my best to answer.

    Get a Handle on Modern Data Preparation!

    Data Preparation for Machine Learning

    Prepare Your Machine Learning Data in Minutes

    …with just a few lines of python code

    Discover how in my new Ebook:
    Data Preparation for Machine Learning

    It provides self-study tutorials with full working code on:
    Feature Selection, RFE, Data Cleaning, Data Transforms, Scaling, Dimensionality Reduction,
    and much more…

    Bring Modern Data Preparation Techniques to
    Your Machine Learning Projects

    See What’s Inside

    Covid Abruzzo Basilicata Calabria Campania Emilia Romagna Friuli Venezia Giulia Lazio Liguria Lombardia Marche Molise Piemonte Puglia Sardegna Sicilia Toscana Trentino Alto Adige Umbria Valle d’Aosta Veneto Italia Agrigento Alessandria Ancona Aosta Arezzo Ascoli Piceno Asti Avellino Bari Barletta-Andria-Trani Belluno Benevento Bergamo Biella Bologna Bolzano Brescia Brindisi Cagliari Caltanissetta Campobasso Carbonia-Iglesias Caserta Catania Catanzaro Chieti Como Cosenza Cremona Crotone Cuneo Enna Fermo Ferrara Firenze Foggia Forlì-Cesena Frosinone Genova Gorizia Grosseto Imperia Isernia La Spezia L’Aquila Latina Lecce Lecco Livorno Lodi Lucca Macerata Mantova Massa-Carrara Matera Messina Milano Modena Monza e della Brianza Napoli Novara Nuoro Olbia-Tempio Oristano Padova Palermo Parma Pavia Perugia Pesaro e Urbino Pescara Piacenza Pisa Pistoia Pordenone Potenza Prato Ragusa Ravenna Reggio Calabria Reggio Emilia Rieti Rimini Roma Rovigo Salerno Medio Campidano Sassari Savona Siena Siracusa Sondrio Taranto Teramo Terni Torino Ogliastra Trapani Trento Treviso Trieste Udine Varese Venezia Verbano-Cusio-Ossola Vercelli Verona Vibo Valentia Vicenza Viterbo

    Hypothesis-Test-for-Comparing-Machine-Learning-Algorithms.png

    Hypothesis Test for Comparing Machine Learning Algorithms

    Machine learning models are chosen based on their mean performance, often calculated using k-fold cross-validation.

    The algorithm with the best mean performance is expected to be better than those algorithms with worse mean performance. But what if the difference in the mean performance is caused by a statistical fluke?

    The solution is to use a statistical hypothesis test to evaluate whether the difference in the mean performance between any two algorithms is real or not.

    In this tutorial, you will discover how to use statistical hypothesis tests for comparing machine learning algorithms.

    After completing this tutorial, you will know:

    • Performing model selection based on the mean model performance can be misleading.
    • The five repeats of two-fold cross-validation with a modified Student’s t-Test is a good practice for comparing machine learning algorithms.
    • How to use the MLxtend machine learning to compare algorithms using a statistical hypothesis test.

    Kick-start your project with my new book Statistics for Machine Learning, including step-by-step tutorials and the Python source code files for all examples.

    Let’s get started.

    Hypothesis Test for Comparing Machine Learning Algorithms

    Hypothesis Test for Comparing Machine Learning Algorithms
    Photo by Frank Shepherd, some rights reserved.

    Tutorial Overview

    This tutorial is divided into three parts; they are:

  • Hypothesis Test for Comparing Algorithms
  • 5×2 Procedure With MLxtend
  • Comparing Classifier Algorithms
  • Hypothesis Test for Comparing Algorithms

    Model selection involves evaluating a suite of different machine learning algorithms or modeling pipelines and comparing them based on their performance.

    The model or modeling pipeline that achieves the best performance according to your performance metric is then selected as your final model that you can then use to start making predictions on new data.

    This applies to regression and classification predictive modeling tasks with classical machine learning algorithms and deep learning. It’s always the same process.

    The problem is, how do you know the difference between two models is real and not just a statistical fluke?

    This problem can be addressed using a statistical hypothesis test.

    One approach is to evaluate each model on the same k-fold cross-validation split of the data (e.g. using the same random number seed to split the data in each case) and calculate a score for each split. This would give a sample of 10 scores for 10-fold cross-validation. The scores can then be compared using a paired statistical hypothesis test because the same treatment (rows of data) was used for each algorithm to come up with each score. The Paired Student’s t-Test could be used.

    A problem with using the Paired Student’s t-Test, in this case, is that each evaluation of the model is not independent. This is because the same rows of data are used to train the data multiple times — actually, each time, except for the time a row of data is used in the hold-out test fold. This lack of independence in the evaluation means that the Paired Student’s t-Test is optimistically biased.

    This statistical test can be adjusted to take the lack of independence into account. Additionally, the number of folds and repeats of the procedure can be configured to achieve a good sampling of model performance that generalizes well to a wide range of problems and algorithms. Specifically two-fold cross-validation with five repeats, so-called 5×2-fold cross-validation.

    This approach was proposed by Thomas Dietterich in his 1998 paper titled “Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms.”

    For more on this topic, see the tutorial:

    Thankfully, we don’t need to implement this procedure ourselves.

    5×2 Procedure With MLxtend

    The MLxtend library by Sebastian Raschka provides an implementation via the paired_ttest_5x2cv() function.

    First, you must install the mlxtend library, for example:


    To use the evaluation, you must first load your dataset, then define the two models that you wish to compare.


    You can then call the paired_ttest_5x2cv() function and pass in your data and models and it will report the t-statistic value and the p-value as to whether the difference in the performance of the two algorithms is significant or not.


    The p-value must be interpreted using an alpha value, which is the significance level that you are willing to accept.

    If the p-value is less or equal to the chosen alpha, we reject the null hypothesis that the models have the same mean performance, which means the difference is probably real. If the p-value is greater than alpha, we fail to reject the null hypothesis that the models have the same mean performance and any observed difference in the mean accuracies is probability a statistical fluke.

    The smaller the alpha value, the better, and a common value is 5 percent (0.05).


    Now that we are familiar with the way to use a hypothesis test to compare algorithms, let’s look at some examples.

    Comparing Classifier Algorithms

    In this section, let’s compare the performance of two machine learning algorithms on a binary classification task, then check if the observed difference is statistically significant or not.

    First, we can use the make_classification() function to create a synthetic dataset with 1,000 samples and 20 input variables.

    The example below creates the dataset and summarizes its shape.


    Running the example creates the dataset and summarizes the number of rows and columns, confirming our expectations.

    We can use this data as the basis for comparing two algorithms.


    We will compare the performance of two linear algorithms on this dataset. Specifically, a logistic regression algorithm and a linear discriminant analysis (LDA) algorithm.

    The procedure I like is to use repeated stratified k-fold cross-validation with 10 folds and three repeats. We will use this procedure to evaluate each algorithm and return and report the mean classification accuracy.

    The complete example is listed below.


    Running the example first reports the mean classification accuracy for each algorithm.

    Your specific results may differ given the stochastic nature of the learning algorithms and evaluation procedure. Try running the example a few times.

    In this case, the results suggest that LDA has better performance if we just look at the mean scores: 89.2 percent for logistic regression and 89.3 percent for LDA.


    A box and whisker plot is also created summarizing the distribution of accuracy scores.

    This plot would support my decision in choosing LDA over LR.

    Box and Whisker Plot of Classification Accuracy Scores for Two Algorithms

    Box and Whisker Plot of Classification Accuracy Scores for Two Algorithms

    Now we can use a hypothesis test to see if the observed results are statistically significant.

    First, we will use the 5×2 procedure to evaluate the algorithms and calculate a p-value and test statistic value.


    We can then interpret the p-value using an alpha of 5 percent.


    Tying this together, the complete example is listed below.


    Running the example, we first evaluate the algorithms before, then report on the result of the statistical hypothesis test.

    Your specific results may differ given the stochastic nature of the learning algorithms and evaluation procedure. Try running the example a few times.

    In this case, we can see that the p-value is about 0.3, which is much larger than 0.05. This leads us to fail to reject the null hypothesis, suggesting that any observed difference between the algorithms is probably not real.

    We could just as easily choose logistic regression or LDA and both would perform about the same on average.

    This highlights that performing model selection based only on the mean performance may not be sufficient.


    Recall that we are reporting performance using a different procedure (3×10 CV) than the procedure used to estimate the performance in the statistical test (5×2 CV). Perhaps results would be different if we looked at scores using five repeats of two-fold cross-validation?

    The example below is updated to report classification accuracy for each algorithm using 5×2 CV.


    Running the example reports the mean accuracy for both algorithms and the results of the statistical test.

    Your specific results may differ given the stochastic nature of the learning algorithms and evaluation procedure. Try running the example a few times.

    In this case, we can see that the difference in the mean performance for the two algorithms is even larger, 89.4 percent vs. 89.0 percent in favor of logistic regression instead of LDA as we saw with 3×10 CV.


    Further Reading

    This section provides more resources on the topic if you are looking to go deeper.

    Tutorials
    Papers
    APIs

    Summary

    In this tutorial, you discovered how to use statistical hypothesis tests for comparing machine learning algorithms.

    Specifically, you learned:

    • Performing model selection based on the mean model performance can be misleading.
    • The five repeats of two-fold cross-validation with a modified Student’s t-Test is a good practice for comparing machine learning algorithms.
    • How to use the MLxtend machine learning to compare algorithms using a statistical hypothesis test.

    Do you have any questions?
    Ask your questions in the comments below and I will do my best to answer.

    Get a Handle on Statistics for Machine Learning!

    Statistical Methods for Machine Learning
    Develop a working understanding of statistics

    …by writing lines of code in python

    Discover how in my new Ebook:
    Statistical Methods for Machine Learning

    It provides self-study tutorials on topics like:
    Hypothesis Tests, Correlation, Nonparametric Stats, Resampling, and much more…

    Discover how to Transform Data into Knowledge

    Skip the Academics. Just Results.

    See What’s Inside

    Covid Abruzzo Basilicata Calabria Campania Emilia Romagna Friuli Venezia Giulia Lazio Liguria Lombardia Marche Molise Piemonte Puglia Sardegna Sicilia Toscana Trentino Alto Adige Umbria Valle d’Aosta Veneto Italia Agrigento Alessandria Ancona Aosta Arezzo Ascoli Piceno Asti Avellino Bari Barletta-Andria-Trani Belluno Benevento Bergamo Biella Bologna Bolzano Brescia Brindisi Cagliari Caltanissetta Campobasso Carbonia-Iglesias Caserta Catania Catanzaro Chieti Como Cosenza Cremona Crotone Cuneo Enna Fermo Ferrara Firenze Foggia Forlì-Cesena Frosinone Genova Gorizia Grosseto Imperia Isernia La Spezia L’Aquila Latina Lecce Lecco Livorno Lodi Lucca Macerata Mantova Massa-Carrara Matera Messina Milano Modena Monza e della Brianza Napoli Novara Nuoro Olbia-Tempio Oristano Padova Palermo Parma Pavia Perugia Pesaro e Urbino Pescara Piacenza Pisa Pistoia Pordenone Potenza Prato Ragusa Ravenna Reggio Calabria Reggio Emilia Rieti Rimini Roma Rovigo Salerno Medio Campidano Sassari Savona Siena Siracusa Sondrio Taranto Teramo Terni Torino Ogliastra Trapani Trento Treviso Trieste Udine Varese Venezia Verbano-Cusio-Ossola Vercelli Verona Vibo Valentia Vicenza Viterbo

    Recent Posts

    Archives