Automated-Machine-Learning-AutoML-Libraries-for-Python.jpg

Automated Machine Learning (AutoML) Libraries for Python

AutoML provides tools to automatically discover good machine learning model pipelines for a dataset with very little user intervention.

It is ideal for domain experts new to machine learning or machine learning practitioners looking to get good results quickly for a predictive modeling task.

Open-source libraries are available for using AutoML methods with popular machine learning libraries in Python, such as the scikit-learn machine learning library.

In this tutorial, you will discover how to use top open-source AutoML libraries for scikit-learn in Python.

After completing this tutorial, you will know:

  • AutoML are techniques for automatically and quickly discovering a well-performing machine learning model pipeline for a predictive modeling task.
  • The three most popular AutoML libraries for Scikit-Learn are Hyperopt-Sklearn, Auto-Sklearn, and TPOT.
  • How to use AutoML libraries to discover well-performing models for predictive modeling tasks in Python.

Let’s get started.

Automated Machine Learning (AutoML) Libraries for Python

Automated Machine Learning (AutoML) Libraries for Python
Photo by Michael Coghlan, some rights reserved.

Tutorial Overview

This tutorial is divided into four parts; they are:

  • Automated Machine Learning
  • Auto-Sklearn
  • Tree-based Pipeline Optimization Tool (TPOT)
  • Hyperopt-Sklearn
  • Automated Machine Learning

    Automated Machine Learning, or AutoML for short, involves the automatic selection of data preparation, machine learning model, and model hyperparameters for a predictive modeling task.

    It refers to techniques that allow semi-sophisticated machine learning practitioners and non-experts to discover a good predictive model pipeline for their machine learning task quickly, with very little intervention other than providing a dataset.

    … the user simply provides data, and the AutoML system automatically determines the approach that performs best for this particular application. Thereby, AutoML makes state-of-the-art machine learning approaches accessible to domain scientists who are interested in applying machine learning but do not have the resources to learn about the technologies behind it in detail.

    — Page ix, Automated Machine Learning: Methods, Systems, Challenges, 2019.

    Central to the approach is defining a large hierarchical optimization problem that involves identifying data transforms and the machine learning models themselves, in addition to the hyperparameters for the models.

    Many companies now offer AutoML as a service, where a dataset is uploaded and a model pipeline can be downloaded or hosted and used via web service (i.e. MLaaS). Popular examples include service offerings from Google, Microsoft, and Amazon.

    Additionally, open-source libraries are available that implement AutoML techniques, focusing on the specific data transforms, models, and hyperparameters used in the search space and the types of algorithms used to navigate or optimize the search space of possibilities, with versions of Bayesian Optimization being the most common.

    There are many open-source AutoML libraries, although, in this tutorial, we will focus on the best-of-breed libraries that can be used in conjunction with the popular scikit-learn Python machine learning library.

    They are: Hyperopt-Sklearn, Auto-Sklearn, and TPOT.

    Did I miss your favorite AutoML library for scikit-learn?
    Let me know in the comments below.

    We will take a closer look at each, providing the basis for you to evaluate and consider which library might be appropriate for your project.

    Auto-Sklearn

    Auto-Sklearn is an open-source Python library for AutoML using machine learning models from the scikit-learn machine learning library.

    It was developed by Matthias Feurer, et al. and described in their 2015 paper titled “Efficient and Robust Automated Machine Learning.”

    … we introduce a robust new AutoML system based on scikit-learn (using 15 classifiers, 14 feature preprocessing methods, and 4 data preprocessing methods, giving rise to a structured hypothesis space with 110 hyperparameters).

    — Efficient and Robust Automated Machine Learning, 2015.

    The first step is to install the Auto-Sklearn library, which can be achieved using pip, as follows:


    Once installed, we can import the library and print the version number to confirm it was installed successfully:


    Running the example prints the version number. Your version number should be the same or higher.


    Next, we can demonstrate using Auto-Sklearn on a synthetic classification task.

    We can define an AutoSklearnClassifier class that controls the search and configure it to run for two minutes (120 seconds) and kill any single model that takes more than 30 seconds to evaluate. At the end of the run, we can report the statistics of the search and evaluate the best performing model on a holdout dataset.

    The complete example is listed below.


    Running the example will take about two minutes, given the hard limit we imposed on the run.

    At the end of the run, a summary is printed showing that 599 models were evaluated and the estimated performance of the final model was 95.6 percent.


    We then evaluate the model on the holdout dataset and see that a classification accuracy of 97 percent was achieved, which is reasonably skillful.


    For more on the Auto-Sklearn library, see:

    Tree-based Pipeline Optimization Tool (TPOT)

    Tree-based Pipeline Optimization Tool, or TPOT for short, is a Python library for automated machine learning.

    TPOT uses a tree-based structure to represent a model pipeline for a predictive modeling problem, including data preparation and modeling algorithms, and model hyperparameters.

    … an evolutionary algorithm called the Tree-based Pipeline Optimization Tool (TPOT) that automatically designs and optimizes machine learning pipelines.

    — Evaluation of a Tree-based Pipeline Optimization Tool for Automating Data Science, 2016.

    The first step is to install the TPOT library, which can be achieved using pip, as follows:


    Once installed, we can import the library and print the version number to confirm it was installed successfully:


    Running the example prints the version number. Your version number should be the same or higher.


    Next, we can demonstrate using TPOT on a synthetic classification task.

    This involves configuring a TPOTClassifier instance with the population size and number of generations for the evolutionary search, as well as the cross-validation procedure and metric used to evaluate models. The algorithm will then run the search procedure and save the best discovered model pipeline to file.

    The complete example is listed below.


    Running the example may take a few minutes, and you will see a progress bar on the command line.

    The accuracy of top-performing models will be reported along the way.

    Your specific results will vary given the stochastic nature of the search procedure.


    In this case, we can see that the top-performing pipeline achieved the mean accuracy of about 92.6 percent.

    The top-performing pipeline is then saved to a file named “tpot_best_model.py“.

    Opening this file, you can see that there is some generic code for loading a dataset and fitting the pipeline. An example is listed below.


    You can then retrieve the code for creating the model pipeline and integrate it into your project.

    For more on TPOT, see the following resources:

    Hyperopt-Sklearn

    HyperOpt is an open-source Python library for Bayesian optimization developed by James Bergstra.

    It is designed for large-scale optimization for models with hundreds of parameters and allows the optimization procedure to be scaled across multiple cores and multiple machines.

    HyperOpt-Sklearn wraps the HyperOpt library and allows for the automatic search of data preparation methods, machine learning algorithms, and model hyperparameters for classification and regression tasks.

    … we introduce Hyperopt-Sklearn: a project that brings the benefits of automatic algorithm configuration to users of Python and scikit-learn. Hyperopt-Sklearn uses Hyperopt to describe a search space over possible configurations of Scikit-Learn components, including preprocessing and classification modules.

    — Hyperopt-Sklearn: Automatic Hyperparameter Configuration for Scikit-Learn, 2014.

    Now that we are familiar with HyperOpt and HyperOpt-Sklearn, let’s look at how to use HyperOpt-Sklearn.

    The first step is to install the HyperOpt library.

    This can be achieved using the pip package manager as follows:


    Next, we must install the HyperOpt-Sklearn library.

    This too can be installed using pip, although we must perform this operation manually by cloning the repository and running the installation from the local files, as follows:


    We can confirm that the installation was successful by checking the version number with the following command:


    This will summarize the installed version of HyperOpt-Sklearn, confirming that a modern version is being used.


    Next, we can demonstrate using Hyperopt-Sklearn on a synthetic classification task.

    We can configure a HyperoptEstimator instance that runs the search, including the classifiers to consider in the search space, the pre-processing steps, and the search algorithm to use. In this case, we will use TPE, or Tree of Parzen Estimators, and perform 50 evaluations.

    At the end of the search, the best performing model pipeline is evaluated and summarized.

    The complete example is listed below.


    Running the example may take a few minutes.

    The progress of the search will be reported and you will see some warnings that you can safely ignore.

    At the end of the run, the best-performing model is evaluated on the holdout dataset and the Pipeline discovered is printed for later use.

    Your specific results may differ given the stochastic nature of the learning algorithm and search process. Try running the example a few times.

    In this case, we can see that the chosen model achieved an accuracy of about 84.8 percent on the holdout test set. The Pipeline involves a SGDClassifier model with no pre-processing.


    The printed model can then be used directly, e.g. the code copy-pasted into another project.

    For more on Hyperopt-Sklearn, see:

    Summary

    In this tutorial, you discovered how to use top open-source AutoML libraries for scikit-learn in Python.

    Specifically, you learned:

    • AutoML are techniques for automatically and quickly discovering a well-performing machine learning model pipeline for a predictive modeling task.
    • The three most popular AutoML libraries for Scikit-Learn are Hyperopt-Sklearn, Auto-Sklearn, and TPOT.
    • How to use AutoML libraries to discover well-performing models for predictive modeling tasks in Python.

    Do you have any questions?
    Ask your questions in the comments below and I will do my best to answer.

    Discover Fast Machine Learning in Python!

    Master Machine Learning With Python
    Develop Your Own Models in Minutes

    …with just a few lines of scikit-learn code

    Learn how in my new Ebook:
    Machine Learning Mastery With Python

    Covers self-study tutorials and end-to-end projects like:
    Loading data, visualization, modeling, tuning, and much more…

    Finally Bring Machine Learning To

    Your Own Projects

    Skip the Academics. Just Results.

    See What’s Inside

    Covid Abruzzo Basilicata Calabria Campania Emilia Romagna Friuli Venezia Giulia Lazio Liguria Lombardia Marche Molise Piemonte Puglia Sardegna Sicilia Toscana Trentino Alto Adige Umbria Valle d’Aosta Veneto Italia Agrigento Alessandria Ancona Aosta Arezzo Ascoli Piceno Asti Avellino Bari Barletta-Andria-Trani Belluno Benevento Bergamo Biella Bologna Bolzano Brescia Brindisi Cagliari Caltanissetta Campobasso Carbonia-Iglesias Caserta Catania Catanzaro Chieti Como Cosenza Cremona Crotone Cuneo Enna Fermo Ferrara Firenze Foggia Forlì-Cesena Frosinone Genova Gorizia Grosseto Imperia Isernia La Spezia L’Aquila Latina Lecce Lecco Livorno Lodi Lucca Macerata Mantova Massa-Carrara Matera Messina Milano Modena Monza e della Brianza Napoli Novara Nuoro Olbia-Tempio Oristano Padova Palermo Parma Pavia Perugia Pesaro e Urbino Pescara Piacenza Pisa Pistoia Pordenone Potenza Prato Ragusa Ravenna Reggio Calabria Reggio Emilia Rieti Rimini Roma Rovigo Salerno Medio Campidano Sassari Savona Siena Siracusa Sondrio Taranto Teramo Terni Torino Ogliastra Trapani Trento Treviso Trieste Udine Varese Venezia Verbano-Cusio-Ossola Vercelli Verona Vibo Valentia Vicenza Viterbo

    Getting-AI-to-Learn-Like-a-Baby-is-Goal-of.jpg

    Getting AI to Learn Like a Baby is Goal of Self-Supervised Learning 

    Scientists are studying how to create AI systems that learn from self-supervision, akin to how babies learn from observing their environment. (Credit: Getty Images) 

    By AI Trends Staff  

    Scientists are working on creating better AI that learns through self-supervision, with the pinnacle being AI that could learn like a baby, based on observation of its environment and interaction with people.  

    This would be an important advance because AI has limitations based on the volume of data required to train machine learning algorithms, and the brittleness of the algorithms when it comes to adjusting to changing circumstances. 

    Yann LeCun, chief AI scientist at Facebook

    “This is the single most important problem to solve in AI today,” stated Yann LeCun, chief AI scientist at Facebook, in an account in the Wall Street Journal. Some early success with self-supervised learning has been seen in the natural language processing used in mobile phones, smart speakers, and customer service bots.   

    Training AI today is time-consuming and expensive. The promise of self-supervised learning is for AI to train itself without the need for external labels attached to the data. Dr. LeCun is now focused on applying self-supervised learning to computer vision, a more complex problem in which computers interpret images such as a person’s face.  

    The next phase, which he thinks is possible in the next decade or two, is to create a machine that can “learn how the world works by watching video , listening to audio, and reading text,” he stated. 

    More than one approach is being tried to help AI learn by itself. One is the neuro-symbolic approach, which combines deep learning and symbolic AI, which represents human knowledge explicitly as facts and rules. IBM is experimenting with this approach in its development of a bot that works alongside human engineers, reading computer logs to look for system failure, understand why a system crashed and offer a remedy. This could increase the pace of scientific discovery, with its ability to spot patterns not otherwise evident, according to Dario Gil, director of IBM Research. “It would help us address huge problems, such as climate change and developing vaccines,” he stated. 

    Child Psychologists Working with Computer Scientists on MESS  

    DARPA is working with the University of California at Berkeley on a research project, Machine Common Sense, funding collaborations between child psychologists and computer scientists. The system is called MESS, for Model-Building, Exploratory, Social Learning System.   

    Alison Gopnik, Professor of Psychology, University of California, Berkeley and the author of “The Philosophical Baby”

    “Human babies are the best learners in the universe. How do they do it? And could we get an AI to do the same?,” queried Alison Gopnik, a professor of psychology at Berkeley and the author of “The Philosophical Baby” and “The Scientist in the Crib,” among other books, in a recent article she wrote for the Wall Street Journal.  

    “Even with a lot of supervised data, AIs can’t make the same kinds of generalizations that human children can,” Gopnik said. “Their knowledge is much narrower and more limited, and they are easily fooled. Current AIs are like children with super-helicopter-tiger moms—programs that hover over the learner dictating whether it is right or wrong at every step. The helicoptered AI children can be very good at learning to do specific things well, but they fall apart when it comes to resilience and creativity. A small change in the learning problem means that they have to start all over again.” 

    The scientists are also experimenting with AI that is motivated by curiosity, which leads to a more resilient learning style, called “active learning” and is a frontier in AI research.  

    The challenge of the DARPA Machine Common Sense program is to design an AI that understands the basic features of the world as well as an 18-month-old. “Some computer scientists are trying to build common sense models into the AIs, though this isn’t easy. But it is even harder to design an AI that can actually learn those models the way that children do,” Dr. Gopnik wrote. “Hybrid systems that combine models with machine learning are one of the most exciting developments at the cutting edge of current AI.” 

    Training AI models on labeled datasets is likely to play a diminished role as self-supervised learning comes into wider use, LeCun said during a session at the virtual International Conference on Learning Representation (ICLR) 2020, which also included Turing Award winner and Canadian computer scientist Yoshua Bengio.  

    The way that self-supervised learning algorithms generate labels from data by exposing relationships between the data’s parts is an advantage.   

    “Most of what we learn as humans and most of what animals learn is in a self-supervised mode, not a reinforcement mode. It’s basically observing the world and interacting with it a little bit, mostly by observation in a test-independent way,” stated LeCun, in an account from VentureBeat.  “This is the type of [learning] that we don’t know how to reproduce with machines.” 

    Bengio was optimistic about the potential for AI to gain from the field of neuroscience, in particular for its explorations of consciousness and conscious processing. Bengio predicted that new studies will clarify the way high-level semantic variables connect with how the brain processes information, including visual information. These variables that humans communicate using language could lead to an entirely new generation of deep learning models, he suggested. 

    “There’s a lot of progress that could be achieved by bringing together things like grounded language learning, where we’re jointly trying to understand a model of the world and how high-level concepts are related to each other,” said Bengio. “Human conscious processing is exploiting assumptions about how the world might change, which can be conveniently implemented as a high-level representation.”  

    Bengio Delivered NeurIPS 2019 Talk on System 2 Self-Supervised Models 

    At the 2019 Conference on Neural Information Processing Systems (NeurIPS 2019), Bengio spoke on this topic in a keynote speech entitled,  “From System 1 Deep Learning to System 2 Deep Learning,” with System 2 referring to self-supervised models.  

    “We want to have machines that understand the world, that build good world models, that understand cause and effect, and can act in the world to acquire knowledge,” he said in an account in TechTalks.  

    The intelligent systems should be able to generalize to different distributions in data, just as children learn to adapt as the environment changes around them. “We need systems that can handle those changes and do continual learning, lifelong learning, and so on,” Bengio stated. “This is a long-standing goal for machine learning, but we haven’t yet built a solution to this.”  

    Read the source articles in the Wall Street Journal, Alison for the Wall Street Journal, in VentureBeat and in TechTalks. 

    Covid Abruzzo Basilicata Calabria Campania Emilia Romagna Friuli Venezia Giulia Lazio Liguria Lombardia Marche Molise Piemonte Puglia Sardegna Sicilia Toscana Trentino Alto Adige Umbria Valle d’Aosta Veneto Italia Agrigento Alessandria Ancona Aosta Arezzo Ascoli Piceno Asti Avellino Bari Barletta-Andria-Trani Belluno Benevento Bergamo Biella Bologna Bolzano Brescia Brindisi Cagliari Caltanissetta Campobasso Carbonia-Iglesias Caserta Catania Catanzaro Chieti Como Cosenza Cremona Crotone Cuneo Enna Fermo Ferrara Firenze Foggia Forlì-Cesena Frosinone Genova Gorizia Grosseto Imperia Isernia La Spezia L’Aquila Latina Lecce Lecco Livorno Lodi Lucca Macerata Mantova Massa-Carrara Matera Messina Milano Modena Monza e della Brianza Napoli Novara Nuoro Olbia-Tempio Oristano Padova Palermo Parma Pavia Perugia Pesaro e Urbino Pescara Piacenza Pisa Pistoia Pordenone Potenza Prato Ragusa Ravenna Reggio Calabria Reggio Emilia Rieti Rimini Roma Rovigo Salerno Medio Campidano Sassari Savona Siena Siracusa Sondrio Taranto Teramo Terni Torino Ogliastra Trapani Trento Treviso Trieste Udine Varese Venezia Verbano-Cusio-Ossola Vercelli Verona Vibo Valentia Vicenza Viterbo

    Toward-a-machine-learning-model-that-can-reason-about-everyday.png

    Toward a machine learning model that can reason about everyday actions | MIT News

    news .mit.edu/sites/default/files/styles/news_article__cover_image__original/public/images/202008/Abstraction-Four.png?itok=c9pTkX_t” />

    The ability to reason abstractly about events as they unfold is a defining feature of human intelligence. We know instinctively that crying and writing are means of communicating, and that a panda falling from a tree and a plane landing are variations on descending. 

    Organizing the world into abstract categories does not come easily to computers, but in recent years researchers have inched closer by training machine learning models on words and images infused with structural information about the world, and how objects, animals, and actions relate. In a new study at the European Conference on Computer Vision this month, researchers unveiled a hybrid language-vision model that can compare and contrast a set of dynamic events captured on video to tease out the high-level concepts connecting them. 

    Their model did as well as or better than humans at two types of visual reasoning tasks — picking the video that conceptually best completes the set, and picking the video that doesn’t fit. Shown videos of a dog barking and a man howling beside his dog, for example, the model completed the set by picking the crying baby from a set of five videos. Researchers replicated their results on two datasets for training AI systems in action recognition: MIT’s Multi-Moments in Time and DeepMind’s Kinetics.

    “We show that you can build abstraction into an AI system to perform ordinary visual reasoning tasks close to a human level,” says the study’s senior author Aude Oliva, a senior research scientist at MIT, co-director of the MIT Quest for Intelligence, and MIT director of the MIT-IBM Watson AI Lab. “A model that can recognize abstract events will give more accurate, logical predictions and be more useful for decision-making.”

    As deep neural networks become expert at recognizing objects and actions in photos and video , researchers have set their sights on the next milestone: abstraction, and training models to reason about what they see. In one approach, researchers have merged the pattern-matching power of deep nets with the logic of symbolic programs to teach a model to interpret complex object relationships in a scene. Here, in another approach, researchers capitalize on the relationships embedded in the meanings of words to give their model visual reasoning power.

    “Language representations allow us to integrate contextual information learned from text databases into our visual models,” says study co-author Mathew Monfort, a research scientist at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL). “Words like ‘running,’ ‘lifting,’ and ‘boxing’ share some common characteristics that make them more closely related to the concept ‘exercising,’ for example, than ‘driving.’ ”

    Using WordNet, a database of word meanings, the researchers mapped the relation of each action-class label in Moments and Kinetics to the other labels in both datasets. Words like “sculpting,” “carving,” and “cutting,” for example, were connected to higher-level concepts like “crafting,” “making art,” and “cooking.” Now when the model recognizes an activity like sculpting, it can pick out conceptually similar activities in the dataset. 

    This relational graph of abstract classes is used to train the model to perform two basic tasks. Given a set of videos, the model creates a numerical representation for each video that aligns with the word representations of the actions shown in the video . An abstraction module then combines the representations generated for each video in the set to create a new set representation that is used to identify the abstraction shared by all the videos in the set.

    To see how the model would do compared to humans, the researchers asked human subjects to perform the same set of visual reasoning tasks online. To their surprise, the model performed as well as humans in many scenarios, sometimes with unexpected results. In a variation on the set completion task, after watching a video of someone wrapping a gift and covering an item in tape, the model suggested a video of someone at the beach burying someone else in the sand. 

    “It’s effectively ‘covering,’ but very different from the visual features of the other clips,” says Camilo Fosco, a PhD student at MIT who is co-first author of the study with PhD student Alex Andonian. “Conceptually it fits, but I had to think about it.”

    Limitations of the model include a tendency to overemphasize some features. In one case, it suggested completing a set of sports videos with a video of a baby and a ball, apparently associating balls with exercise and competition.

    A deep learning model that can be trained to “think” more abstractly may be capable of learning with fewer data, say researchers. Abstraction also paves the way toward higher-level, more human-like reasoning.

    “One hallmark of human cognition is our ability to describe something in relation to something else — to compare and to contrast,” says Oliva. “It’s a rich and efficient way to learn that could eventually lead to machine learning models that can understand analogies and are that much closer to communicating intelligently with us.”

    Other authors of the study are Allen Lee from MIT, Rogerio Feris from IBM, and Carl Vondrick from Columbia University.

    Covid Abruzzo Basilicata Calabria Campania Emilia Romagna Friuli Venezia Giulia Lazio Liguria Lombardia Marche Molise Piemonte Puglia Sardegna Sicilia Toscana Trentino Alto Adige Umbria Valle d’Aosta Veneto Italia Agrigento Alessandria Ancona Aosta Arezzo Ascoli Piceno Asti Avellino Bari Barletta-Andria-Trani Belluno Benevento Bergamo Biella Bologna Bolzano Brescia Brindisi Cagliari Caltanissetta Campobasso Carbonia-Iglesias Caserta Catania Catanzaro Chieti Como Cosenza Cremona Crotone Cuneo Enna Fermo Ferrara Firenze Foggia Forlì-Cesena Frosinone Genova Gorizia Grosseto Imperia Isernia La Spezia L’Aquila Latina Lecce Lecco Livorno Lodi Lucca Macerata Mantova Massa-Carrara Matera Messina Milano Modena Monza e della Brianza Napoli Novara Nuoro Olbia-Tempio Oristano Padova Palermo Parma Pavia Perugia Pesaro e Urbino Pescara Piacenza Pisa Pistoia Pordenone Potenza Prato Ragusa Ravenna Reggio Calabria Reggio Emilia Rieti Rimini Roma Rovigo Salerno Medio Campidano Sassari Savona Siena Siracusa Sondrio Taranto Teramo Terni Torino Ogliastra Trapani Trento Treviso Trieste Udine Varese Venezia Verbano-Cusio-Ossola Vercelli Verona Vibo Valentia Vicenza Viterbo

    Recent Posts

    Archives

    wpChatIcon