Optimization

Featured Image

Combined Algorithm Selection and Hyperparameter Optimization (CASH Optimization)

Machine learning model selection and configuration may be the biggest challenge in applied machine learning.

Controlled experiments must be performed in order to discover what works best for a given classification or regression predictive modeling task. This can feel overwhelming given the large number of data preparation schemes, learning algorithms, and model hyperparameters that could be considered.

The common approach is to use a shortcut, such as using a popular algorithm or testing a small number of algorithms with default hyperparameters.

A modern alternative is to consider the selection of data preparation, learning algorithm, and algorithm hyperparameters one large global optimization problem. This characterization is generally referred to as Combined Algorithm Selection and Hyperparameter Optimization, or “CASH Optimization” for short.

In this post, you will discover the challenge of machine learning model selection and the modern solution referred to CASH Optimization.

After reading this post, you will know:

  • The challenge of machine learning model and hyperparameter selection.
  • The shortcuts of using popular models or making a series of sequential decisions.
  • The characterization of Combined Algorithm Selection and Hyperparameter Optimization that underlies modern AutoML.

Let’s get started.

Combined Algorithm Selection and Hyperparameter Optimization (CASH Optimization)

Combined Algorithm Selection and Hyperparameter Optimization (CASH Optimization)
Photo by Bernard Spragg. NZ, some rights reserved.

Overview

This tutorial is divided into three parts; they are:

  • Challenge of Model and Hyperparameter Selection
  • Solutions to Model and Hyperparameter Selection
  • Combined Algorithm Selection and Hyperparameter Optimization

    • Challenge of Model and Hyperparameter Selection

    There is no definitive mapping of machine learning algorithms to predictive modeling tasks.

    We cannot look at a dataset and know the best algorithm to use, let alone the best data transforms to use to prepare the data or the best configuration for a given model.

    Instead, we must use controlled experiments to discover what works best for a given dataset.

    As such, applied machine learning is an empirical discipline. It is engineering and art more than science.

    The problem is that there are tens, if not hundreds, of machine learning algorithms to choose from. Each algorithm may have up to tens of hyperparameters to be configured.

    To a beginner, the scope of the problem is overwhelming.

    • Where do you start?
    • What do you start with?
    • When do you discard a model?
    • When do you double down on a model?

    There are a few standard solutions to this problem adopted by most practitioners, experienced and otherwise.

    Solutions to Model and Hyperparameter Selection

    Let’s look at two of the most common short-cuts to this problem of selecting data transforms, machine learning models, and model hyperparameters.

    Use a Popular Algorithm

    One approach is to use a popular machine learning algorithm.

    It can be challenging to make the right choice when faced with these degrees of freedom, leaving many users to select algorithms based on reputation or intuitive appeal, and/or to leave hyperparameters set to default values. Of course, this approach can yield performance far worse than that of the best method and hyperparameter settings.

    — Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms, 2012.

    For example, if it seems like everyone is talking about “random forest,” then random forest becomes the right algorithm for all classification and regression problems you encounter, and you limit the experimentation to the hyperparameters of the random forest algorithm.

    • Short-Cut #1: Use a popular algorithm like “random forest” or “xgboost“.

    Random forest indeed performs well on a wide range of prediction tasks. But we cannot know if it will be good or even best for a given dataset. The risk is that we may be able to achieve better results with a much simpler linear model.

    A workaround might be to test a range of popular algorithms, leading into the next shortcut.

    Sequentially Test Transforms, Models, and Hyperparameters

    Another approach is to approach the problem as a series of sequential decisions.

    For example, review the data and select data transforms that make data more Gaussian, remove outliers, etc. Then test a suite of algorithms with default hyperparameters and select one or a few that perform well. Then tune the hyperparameters of those top-performing models.

    • Short-Cut #2: Sequentially select data transforms, models, and model hyperparameters.

    This is the approach that I recommend for getting good results quickly; for example:

    This short-cut too can be effective and reduces the likelihood of missing an algorithm that performs well on your dataset. The downside here is more subtle and impacts you if you are seeking great or excellent results rather than merely good results quickly.

    The risk is selecting data transforms prior to selecting models might mean that you miss the data preparation sequence that gets the most out of an algorithm.

    Similarly, selecting a model or subset of models prior to selecting model hyperparameters means that you might be missing a model with hyperparameters other than the default values that performs better than any of the subset of models selected and their subsequent configurations.

    Two important problems in AutoML are that (1) no single machine learning method performs best on all datasets and (2) some machine learning methods (e.g., non-linear SVMs) crucially rely on hyperparameter optimization.

    — Page 115, Automated Machine Learning: Methods, Systems, Challenges, 2019.

    A workaround might be to spot check good or well-performing configurations of each algorithm as part of the algorithm spot check. This is only a partial solution.

    There is a better approach.

    Combined Algorithm Selection and Hyperparameter Optimization

    Selecting a data preparation pipeline, machine learning model, and model hyperparameters is a search problem.

    The possible choices at each step define a search space, and a single combination represents a point in that space that can be evaluated with a dataset.

    Navigating the search space efficiently is referred to as global optimization.

    This has been well understood for a long time in the field of machine learning, although perhaps tacitly, with focus typically on one element of the problem, such as hyperparameter optimization.

    The important insight is that there are dependencies between each step, which influences the size and structure of the search space.

    … [the problem] can be viewed as a single hierarchical hyperparameter optimization problem, in which even the choice of algorithm itself is considered a hyperparameter.

    — Page 82, Automated Machine Learning: Methods, Systems, Challenges, 2019.

    This requires that the data preparation and machine learning model, along with the model hyperparameters, must form the scope of the optimization problem and that the optimization algorithm must be aware of the dependencies between.

    This is a challenging global optimization problem, notably because of the dependencies, but also because estimating the performance of a machine learning model on a dataset is stochastic, resulting in a noisy distribution of performance scores (e.g. via repeated k-fold cross-validation).

    … the combined space of learning algorithms and their hyperparameters is very challenging to search: the response function is noisy and the space is high dimensional, involves both categorical and continuous choices, and contains hierarchical dependencies (e.g., the hyperparameters of a learning algorithm are only meaningful if that algorithm is chosen; the algorithm choices in an ensemble method are only meaningful if that ensemble method is chosen; etc).

    — Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms, 2012.

    This challenge was perhaps best characterized by Chris Thornton, et al. in their 2013 paper titled “Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms.” In the paper, they refer to this problem as “Combined Algorithm Selection And Hyperparameter Optimization,” or “CASH Optimization” for short.

    … a natural challenge for machine learning: given a dataset, to automatically and simultaneously choose a learning algorithm and set its hyperparameters to optimize empirical performance. We dub this the combined algorithm selection and hyperparameter optimization problem (short: CASH).

    — Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms, 2012.

    This characterization is also sometimes referred to as “Full Model Selection,” or FMS for short.

    The FMS problem consists of the following: given a pool of preprocessing methods, feature selection and learning algorithms, select the combination of these that obtains the lowest classification error for a given data set. This task also includes the selection of hyperparameters for the considered methods, resulting in a vast search space that is well suited for stochastic optimization techniques.

    — Particle Swarm Model Selection, 2009.

    Thornton, et al. proceeded to use global optimization algorithms that are aware of the dependencies, so-called sequential global optimization algorithms, such as specific versions of Bayesian Optimization. They then proceeded to implement their approach for the WEKA machine learning workbench, called the AutoWEKA Projects.

    A promising approach is Bayesian Optimization, and in particular Sequential Model-Based Optimization (SMBO), a versatile stochastic optimization framework that can work with both categorical and continuous hyperparameters, and that can exploit hierarchical structure stemming from conditional parameters.

    — Page 85, Automated Machine Learning: Methods, Systems, Challenges, 2019.

    This now provides the dominant paradigm for a field of study referred to as “Automated Machine Learning,” or AutoML for short. AutoML is concerned with providing tools that allow practitioners with modest technical skill to quickly find effective solutions to machine learning tasks, such as classification and regression predictive modeling.

    AutoML aims to provide effective off-the-shelf learning systems to free experts and non-experts alike from the tedious and time-consuming tasks of selecting the right algorithm for a dataset at hand, along with the right preprocessing method and the various hyperparameters of all involved components.

    — Page 136,Automated Machine Learning: Methods, Systems, Challenges, 2019.

    AutoML techniques are provided by machine learning libraries and increasingly as services, so-called machine learning as a service, or MLaaS for short.

    Further Reading

    This section provides more resources on the topic if you are looking to go deeper.

    Papers
    Books
    Articles

    Summary

    In this post, you discovered the challenge of machine learning model selection and the modern solution referred to as CASH Optimization.

    Specifically, you learned:

    • The challenge of machine learning model and hyperparameter selection.
    • The shortcuts of using popular models or making a series of sequential decisions.
    • The characterization of Combined Algorithm Selection and Hyperparameter Optimization that underlies modern AutoML.

    Do you have any questions?
    Ask your questions in the comments below and I will do my best to answer.

    Discover Fast Machine Learning in Python!

    Master Machine Learning With Python
    Develop Your Own Models in Minutes

    …with just a few lines of scikit-learn code

    Learn how in my new Ebook:
    Machine Learning Mastery With Python

    Covers self-study tutorials and end-to-end projects like:
    Loading data, visualization, modeling, tuning, and much more…

    Finally Bring Machine Learning To

    Your Own Projects

    Skip the Academics. Just Results.

    See What’s Inside

    Covid Abruzzo Basilicata Calabria Campania Emilia Romagna Friuli Venezia Giulia Lazio Liguria Lombardia Marche Molise Piemonte Puglia Sardegna Sicilia Toscana Trentino Alto Adige Umbria Valle d’Aosta Veneto Italia Agrigento Alessandria Ancona Aosta Arezzo Ascoli Piceno Asti Avellino Bari Barletta-Andria-Trani Belluno Benevento Bergamo Biella Bologna Bolzano Brescia Brindisi Cagliari Caltanissetta Campobasso Carbonia-Iglesias Caserta Catania Catanzaro Chieti Como Cosenza Cremona Crotone Cuneo Enna Fermo Ferrara Firenze Foggia Forlì-Cesena Frosinone Genova Gorizia Grosseto Imperia Isernia La Spezia L’Aquila Latina Lecce Lecco Livorno Lodi Lucca Macerata Mantova Massa-Carrara Matera Messina Milano Modena Monza e della Brianza Napoli Novara Nuoro Olbia-Tempio Oristano Padova Palermo Parma Pavia Perugia Pesaro e Urbino Pescara Piacenza Pisa Pistoia Pordenone Potenza Prato Ragusa Ravenna Reggio Calabria Reggio Emilia Rieti Rimini Roma Rovigo Salerno Medio Campidano Sassari Savona Siena Siracusa Sondrio Taranto Teramo Terni Torino Ogliastra Trapani Trento Treviso Trieste Udine Varese Venezia Verbano-Cusio-Ossola Vercelli Verona Vibo Valentia Vicenza Viterbo

    compulor project - September 24, 2020 - International, Machine Learning - , , , , , - No Comments
    Featured Image

    Hyperparameter Optimization With Random Search and Grid Search

    Machine learning models have hyperparameters that you must set in order to customize the model to your dataset.

    Often the general effects of hyperparameters on a model are known, but how to best set a hyperparameter and combinations of interacting hyperparameters for a given dataset is challenging. There are often general heuristics or rules of thumb for configuring hyperparameters.

    A better approach is to objectively search different values for model hyperparameters and choose a subset that results in a model that achieves the best performance on a given dataset. This is called hyperparameter optimization or hyperparameter tuning and is available in the scikit-learn Python machine learning library. The result of a hyperparameter optimization is a single set of well-performing hyperparameters that you can use to configure your model.

    In this tutorial, you will discover hyperparameter optimization for machine learning in Python.

    After completing this tutorial, you will know:

    • Hyperparameter optimization is required to get the most out of your machine learning models.
    • How to configure random and grid search hyperparameter optimization for classification tasks.
    • How to configure random and grid search hyperparameter optimization for regression tasks.

    Let’s get started.

    Hyperparameter Optimization With Random Search and Grid Search

    Hyperparameter Optimization With Random Search and Grid Search
    Photo by James St. John, some rights reserved.

    Tutorial Overview

    This tutorial is divided into five parts; they are:

  • Model Hyperparameter Optimization
  • Hyperparameter Optimization Scikit-Learn API
  • Hyperparameter Optimization for Classification
  • Random Search for Classification
  • Grid Search for Classification
  • Hyperparameter Optimization for Regression
  • Random Search for Regression
  • Grid Search for Regression
  • Common Questions About Hyperparameter Optimization

    • Model Hyperparameter Optimization

    Machine learning models have hyperparameters.

    Hyperparameters are points of choice or configuration that allow a machine learning model to be customized for a specific task or dataset.

    • Hyperparameter: Model configuration argument specified by the developer to guide the learning process for a specific dataset.

    Machine learning models also have parameters, which are the internal coefficients set by training or optimizing the model on a training dataset.

    Parameters are different from hyperparameters. Parameters are learned automatically; hyperparameters are set manually to help guide the learning process.

    For more on the difference between parameters and hyperparameters, see the tutorial:

    Typically a hyperparameter has a known effect on a model in the general sense, but it is not clear how to best set a hyperparameter for a given dataset. Further, many machine learning models have a range of hyperparameters and they may interact in nonlinear ways.

    As such, it is often required to search for a set of hyperparameters that result in the best performance of a model on a dataset. This is called hyperparameter optimization, hyperparameter tuning, or hyperparameter search.

    An optimization procedure involves defining a search space. This can be thought of geometrically as an n-dimensional volume, where each hyperparameter represents a different dimension and the scale of the dimension are the values that the hyperparameter may take on, such as real-valued, integer-valued, or categorical.

    • Search Space: Volume to be searched where each dimension represents a hyperparameter and each point represents one model configuration.

    A point in the search space is a vector with a specific value for each hyperparameter value. The goal of the optimization procedure is to find a vector that results in the best performance of the model after learning, such as maximum accuracy or minimum error.

    A range of different optimization algorithms may be used, although two of the simplest and most common methods are random search and grid search.

    • Random Search. Define a search space as a bounded domain of hyperparameter values and randomly sample points in that domain.
    • Grid Search. Define a search space as a grid of hyperparameter values and evaluate every position in the grid.

    Grid search is great for spot-checking combinations that are known to perform well generally. Random search is great for discovery and getting hyperparameter combinations that you would not have guessed intuitively, although it often requires more time to execute.

    More advanced methods are sometimes used, such as Bayesian Optimization and Evolutionary Optimization.

    Now that we are familiar with hyperparameter optimization, let’s look at how we can use this method in Python.

    Hyperparameter Optimization Scikit-Learn API

    The scikit-learn Python open-source machine learning library provides techniques to tune model hyperparameters.

    Specifically, it provides the RandomizedSearchCV for random search and GridSearchCV for grid search. Both techniques evaluate models for a given hyperparameter vector using cross-validation, hence the “CV” suffix of each class name.

    Both classes require two arguments. The first is the model that you are optimizing. This is an instance of the model with values of hyperparameters set that you do not want to optimize. The second is the search space. This is defined as a dictionary where the names are the hyperparameter arguments to the model and the values are discrete values or a distribution of values to sample in the case of a random search.


    Both classes provide a “cv” argument that allows either an integer number of folds to be specified, e.g. 5, or a configured cross-validation object. I recommend defining and specifying a cross-validation object to gain more control over model evaluation and make the evaluation procedure obvious and explicit.

    In the case of classification tasks, I recommend using the RepeatedStratifiedKFold class, and for regression tasks, I recommend using the RepeatedKFold with an appropriate number of folds and repeats, such as 10 folds and three repeats.


    Both hyperparameter optimization classes also provide a “scoring” argument that takes a string indicating the metric to optimize.

    The metric must be maximizing, meaning better models result in larger scores. For classification, this may be ‘accuracy‘. For regression, this is a negative error measure, such as ‘neg_mean_absolute_error‘ for a negative version of the mean absolute error, where values closer to zero represent less prediction error by the model.


    You can see a list of build-in scoring metrics here:

    Finally, the search can be made parallel, e.g. use all of the CPU cores by specifying the “n_jobs” argument as an integer with the number of cores in your system, e.g. 8. Or you can set it to be -1 to automatically use all of the cores in your system.


    Once defined, the search is performed by calling the fit() function and providing a dataset used to train and evaluate model hyperparameter combinations using cross-validation.


    Running the search may take minutes or hours, depending on the size of the search space and the speed of your hardware. You’ll often want to tailor the search to how much time you have rather than the possibility of what could be searched.

    At the end of the search, you can access all of the results via attributes on the class. Perhaps the most important attributes are the best score observed and the hyperparameters that achieved the best score.


    Once you know the set of hyperparameters that achieve the best result, you can then define a new model, set the values of each hyperparameter, then fit the model on all available data. This model can then be used to make predictions on new data.

    Now that we are familiar with the hyperparameter optimization API in scikit-learn, let’s look at some worked examples.

    Hyperparameter Optimization for Classification

    In this section, we will use hyperparameter optimization to discover a well-performing model configuration for the sonar dataset.

    The sonar dataset is a standard machine learning dataset comprising 208 rows of data with 60 numerical input variables and a target variable with two class values, e.g. binary classification.

    Using a test harness of repeated stratified 10-fold cross-validation with three repeats, a naive model can achieve an accuracy of about 53 percent. A top-performing model can achieve accuracy on this same test harness of about 88 percent. This provides the bounds of expected performance on this dataset.

    The dataset involves predicting whether sonar returns indicate a rock or simulated mine.

    No need to download the dataset; we will download it automatically as part of our worked examples.

    The example below downloads the dataset and summarizes its shape.


    Running the example downloads the dataset and splits it into input and output elements. As expected, we can see that there are 208 rows of data with 60 input variables.


    Next, let’s use random search to find a good model configuration for the sonar dataset.

    To keep things simple, we will focus on a linear model, the logistic regression model, and the common hyperparameters tuned for this model.

    Random Search for Classification

    In this section, we will explore hyperparameter optimization of the logistic regression model on the sonar dataset.

    First, we will define the model that will be optimized and use default values for the hyperparameters that will not be optimized.


    We will evaluate model configurations using repeated stratified k-fold cross-validation with three repeats and 10 folds.


    Next, we can define the search space.

    This is a dictionary where names are arguments to the model and values are distributions from which to draw samples. We will optimize the solver, the penalty, and the C hyperparameters of the model with discrete distributions for the solver and penalty type and a log-uniform distribution from 1e-5 to 100 for the C value.

    Log-uniform is useful for searching penalty values as we often explore values at different orders of magnitude, at least as a first step.


    Next, we can define the search procedure with all of these elements.

    Importantly, we must set the number of iterations or samples to draw from the search space via the “n_iter” argument. In this case, we will set it to 500.


    Finally, we can perform the optimization and report the results.


    Tying this together, the complete example is listed below.


    Running the example may take a minute. It is fast because we are using a small search space and a fast model to fit and evaluate. You may see some warnings during the optimization for invalid configuration combinations. These can be safely ignored.

    At the end of the run, the best score and hyperparameter configuration that achieved the best performance are reported.

    Your specific results will vary given the stochastic nature of the optimization procedure. Try running the example a few times.

    In this case, we can see that the best configuration achieved an accuracy of about 78.9 percent, which is fair, and the specific values for the solver, penalty, and C hyperparameters used to achieve that score.


    Next, let’s use grid search to find a good model configuration for the sonar dataset.

    Grid Search for Classification

    Using the grid search is much like using the random search for classification.

    The main difference is that the search space must be a discrete grid to be searched. This means that instead of using a log-uniform distribution for C, we can specify discrete values on a log scale.


    Additionally, the GridSearchCV class does not take a number of iterations, as we are only evaluating combinations of hyperparameters in the grid.


    Tying this together, the complete example of grid searching logistic regression configurations for the sonar dataset is listed below.


    Running the example may take a moment. It is fast because we are using a small search space and a fast model to fit and evaluate. Again, you may see some warnings during the optimization for invalid configuration combinations. These can be safely ignored.

    At the end of the run, the best score and hyperparameter configuration that achieved the best performance are reported.

    Your specific results will vary given the stochastic nature of the optimization procedure. Try running the example a few times.

    In this case, we can see that the best configuration achieved an accuracy of about 78.2% which is also fair and the specific values for the solver, penalty and C hyperparameters used to achieve that score. Interestingly, the results are very similar to those found via the random search.


    Hyperparameter Optimization for Regression

    In this section we will use hyper optimization to discover a top-performing model configuration for the auto insurance dataset.

    The auto insurance dataset is a standard machine learning dataset comprising 63 rows of data with 1 numerical input variable and a numerical target variable.

    Using a test harness of repeated stratified 10-fold cross-validation with 3 repeats, a naive model can achieve a mean absolute error (MAE) of about 66. A top performing model can achieve a MAE on this same test harness of about 28. This provides the bounds of expected performance on this dataset.

    The dataset involves predicting the total amount in claims (thousands of Swedish Kronor) given the number of claims for different geographical regions.

    No need to download the dataset, we will download it automatically as part of our worked examples.

    The example below downloads the dataset and summarizes its shape.


    Running the example downloads the dataset and splits it into input and output elements. As expected, we can see that there are 63 rows of data with 1 input variable.


    Next, we can use hyperparameter optimization to find a good model configuration for the auto insurance dataset.

    To keep things simple, we will focus on a linear model, the linear regression model and the common hyperparameters tuned for this model.

    Random Search for Regression

    Configuring and using the random search hyperparameter optimization procedure for regression is much like using it for classification.

    In this case, we will configure the important hyperparameters of the linear regression implementation, including the solver, alpha, fit_intercept, and normalize.

    We will use a discrete distribution of values in the search space for all except the “alpha” argument which is a penalty term, in which case we will use a log-uniform distribution as we did in the previous section for the “C” argument of logistic regression.


    The main difference in regression compared to classification is the choice of the scoring method.

    For regression, performance is often measured using an error, which is minimized, with zero representing a model with perfect skill. The hyperparameter optimization procedures in scikit-learn assume a maximizing score. Therefore a version of each error metric is provided that is made negative.

    This means that large positive errors become large negative errors, good performance are small negative values close to zero and perfect skill is zero.

    The sign of the negative MAE can be ignored when interpreting the result.

    In this case we will mean absolute error (MAE) and a maximizing version of this error is available by setting the “scoring” argument to “neg_mean_absolute_error“.


    Tying this together, the complete example is listed below.


    Running the example may take a moment. It is fast because we are using a small search space and a fast model to fit and evaluate. You may see some warnings during the optimization for invalid configuration combinations. These can be safely ignored.

    At the end of the run, the best score and hyperparameter configuration that achieved the best performance are reported.

    Your specific results will vary given the stochastic nature of the optimization procedure. Try running the example a few times.

    In this case, we can see that the best configuration achieved a MAE of about 29.2, which is very close to the best performance on the model. We can then see the specific hyperparameter values that achieved this result.


    Next, let’s use grid search to find a good model configuration for the auto insurance dataset.

    Grid Search for Regression

    As a grid search, we cannot define a distribution to sample and instead must define a discrete grid of hyperparameter values. As such, we will specify the “alpha” argument as a range of values on a log-10 scale.


    Grid search for regression requires that the “scoring” be specified, much as we did for random search.

    In this case, we will again use the negative MAE scoring function.


    Tying this together, the complete example of grid searching linear regression configurations for the auto insurance dataset is listed below.


    Running the example may take a minute. It is fast because we are using a small search space and a fast model to fit and evaluate. Again, you may see some warnings during the optimization for invalid configuration combinations. These can be safely ignored.

    At the end of the run, the best score and hyperparameter configuration that achieved the best performance are reported.

    Your specific results will vary given the stochastic nature of the optimization procedure. Try running the example a few times.

    In this case, we can see that the best configuration achieved a MAE of about 29.2, which is nearly identical to what we achieved with the random search in the previous section. Interestingly, the hyperparameters are also nearly identical, which is good confirmation.


    Common Questions About Hyperparameter Optimization

    This section addresses some common questions about hyperparameter optimization.

    How to Choose Between Random and Grid Search?

    Choose the method based on your needs. I recommend starting with grid and doing a random search if you have the time.

    Grid search is appropriate for small and quick searches of hyperparameter values that are known to perform well generally.

    Random search is appropriate for discovering new hyperparameter values or new combinations of hyperparameters, often resulting in better performance, although it may take more time to complete.

    How to Speed-Up Hyperparameter Optimization?

    Ensure that you set the “n_jobs” argument to the number of cores on your machine.

    After that, more suggestions include:

    • Evaluate on a smaller sample of your dataset.
    • Explore a smaller search space.
    • Use fewer repeats and/or folds for cross-validation.
    • Execute the search on a faster machine, such as AWS EC2.
    • Use an alternate model that is faster to evaluate.

    How to Choose Hyperparameters to Search?

    Most algorithms have a subset of hyperparameters that have the most influence over the search procedure.

    These are listed in most descriptions of the algorithm. For example, here are some algorithms and their most important hyperparameters:

    If you are unsure:

    • Review papers that use the algorithm to get ideas.
    • Review the API and algorithm documentation to get ideas.
    • Search all hyperparameters.

    How to Use Best-Performing Hyperparameters?

    Define a new model and set the hyperparameter values of the model to the values found by the search.

    Then fit the model on all available data and use the model to start making predictions on new data.

    This is called preparing a final model. See more here:

    How to Make a Prediction?

    First, fit a final model (previous question).

    Then call the predict() function to make a prediction.

    For examples of making a prediction with a final model, see the tutorial:

    Do you have another question about hyperparameter optimization?
    Let me know in the comments below.

    Further Reading

    This section provides more resources on the topic if you are looking to go deeper.

    Tutorials
    APIs
    Articles

    Summary

    In this tutorial, you discovered hyperparameter optimization for machine learning in Python.

    Specifically, you learned:

    • Hyperparameter optimization is required to get the most out of your machine learning models.
    • How to configure random and grid search hyperparameter optimization for classification tasks.
    • How to configure random and grid search hyperparameter optimization for regression tasks.

    Do you have any questions?
    Ask your questions in the comments below and I will do my best to answer.

    Discover Fast Machine Learning in Python!

    Master Machine Learning With Python
    Develop Your Own Models in Minutes

    …with just a few lines of scikit-learn code

    Learn how in my new Ebook:
    Machine Learning Mastery With Python

    Covers self-study tutorials and end-to-end projects like:
    Loading data, visualization, modeling, tuning, and much more…

    Finally Bring Machine Learning To

    Your Own Projects

    Skip the Academics. Just Results.

    See What’s Inside

    Covid Abruzzo Basilicata Calabria Campania Emilia Romagna Friuli Venezia Giulia Lazio Liguria Lombardia Marche Molise Piemonte Puglia Sardegna Sicilia Toscana Trentino Alto Adige Umbria Valle d’Aosta Veneto Italia Agrigento Alessandria Ancona Aosta Arezzo Ascoli Piceno Asti Avellino Bari Barletta-Andria-Trani Belluno Benevento Bergamo Biella Bologna Bolzano Brescia Brindisi Cagliari Caltanissetta Campobasso Carbonia-Iglesias Caserta Catania Catanzaro Chieti Como Cosenza Cremona Crotone Cuneo Enna Fermo Ferrara Firenze Foggia Forlì-Cesena Frosinone Genova Gorizia Grosseto Imperia Isernia La Spezia L’Aquila Latina Lecce Lecco Livorno Lodi Lucca Macerata Mantova Massa-Carrara Matera Messina Milano Modena Monza e della Brianza Napoli Novara Nuoro Olbia-Tempio Oristano Padova Palermo Parma Pavia Perugia Pesaro e Urbino Pescara Piacenza Pisa Pistoia Pordenone Potenza Prato Ragusa Ravenna Reggio Calabria Reggio Emilia Rieti Rimini Roma Rovigo Salerno Medio Campidano Sassari Savona Siena Siracusa Sondrio Taranto Teramo Terni Torino Ogliastra Trapani Trento Treviso Trieste Udine Varese Venezia Verbano-Cusio-Ossola Vercelli Verona Vibo Valentia Vicenza Viterbo

    compulor project - September 24, 2020 - International, Machine Learning - , , , , - No Comments

    Recent Posts

    Archives