Data-systems-that-learn-to-be-better-MIT-News.jpg

Data systems that learn to be better | MIT News

news .mit.edu/sites/default/files/styles/news_article__cover_image__original/public/images/202008/data-center-00.jpg?itok=6NXbqabZ” />

Big data has gotten really, really big: By 2025, all the world’s data will add up to an estimated 175 trillion gigabytes. For a visual, if you stored that amount of data on DVDs, it would stack up tall enough to circle the Earth 222 times. 

One of the biggest challenges in computing is handling this onslaught of information while still being able to efficiently store and process it. A team from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) believe that the answer rests with something called “instance-optimized systems.”  

Traditional storage and database systems are designed to work for a wide range of applications because of how long it can take to build them — months or, often, several years. As a result, for any given workload such systems provide performance that is good, but usually not the best. Even worse, they sometimes require administrators to painstakingly tune the system by hand to provide even reasonable performance. 

In contrast, the goal of instance-optimized systems is to build systems that optimize and partially re-organize themselves for the data they store and the workload they serve. 

“It’s like building a database system for every application from scratch, which is not economically feasible with traditional system designs,” says MIT Professor Tim Kraska. 

As a first step toward this vision, Kraska and colleagues developed Tsunami and Bao. Tsunami uses machine learning to automatically re-organize a dataset’s storage layout based on the types of queries that its users make. Tests show that it can run queries up to 10 times faster than state-of-the-art systems. What’s more, its datasets can be organized via a series of “learned indexes” that are up to 100 times smaller than the indexes used in traditional systems. 

Kraska has been exploring the topic of learned indexes for several years, going back to his influential work with colleagues at Google in 2017. 

Harvard University Professor Stratos Idreos, who was not involved in the Tsunami project, says that a unique advantage of learned indexes is their small size, which, in addition to space savings, brings substantial performance improvements.

“I think this line of work is a paradigm shift that’s going to impact system design long-term,” says Idreos. “I expect approaches based on models will be one of the core components at the heart of a new wave of adaptive systems.”

Bao, meanwhile, focuses on improving the efficiency of query optimization through machine learning. A query optimizer rewrites a high-level declarative query to a query plan, which can actually be executed over the data to compute the result to the query. However, often there exists more than one query plan to answer any query; picking the wrong one can cause a query to take days to compute the answer, rather than seconds. 

Traditional query optimizers take years to build, are very hard to maintain, and, most importantly, do not learn from their mistakes. Bao is the first learning-based approach to query optimization that has been fully integrated into the popular database management system PostgreSQL. Lead author Ryan Marcus, a postdoc in Kraska’s group, says that Bao produces query plans that run up to 50 percent faster than those created by the PostgreSQL optimizer, meaning that it could help to significantly reduce the cost of cloud services, like Amazon’s Redshift, that are based on PostgreSQL.

By fusing the two systems together, Kraska hopes to build the first instance-optimized database system that can provide the best possible performance for each individual application without any manual tuning. 

The goal is to not only relieve developers from the daunting and laborious process of tuning database systems, but to also provide performance and cost benefits that are not possible with traditional systems.

Traditionally, the systems we use to store data are limited to only a few storage options and, because of it, they cannot provide the best possible performance for a given application. What Tsunami can do is dynamically change the structure of the data storage based on the kinds of queries that it receives and create new ways to store data, which are not feasible with more traditional approaches.

Johannes Gehrke, a managing director at Microsoft Research who also heads up machine learning efforts for Microsoft Teams, says that his work opens up many interesting applications, such as doing so-called “multidimensional queries” in main-memory data warehouses. Harvard’s Idreos also expects the project to spur further work on how to maintain the good performance of such systems when new data and new kinds of queries arrive.

Bao is short for “bandit optimizer,” a play on words related to the so-called “multi-armed bandit” analogy where a gambler tries to maximize their winnings at multiple slot machines that have different rates of return. The multi-armed bandit problem is commonly found in any situation that has tradeoffs between exploring multiple different options, versus exploiting a single option — from risk optimization to A/B testing.

“Query optimizers have been around for years, but they often make mistakes, and usually they don’t learn from them,” says Kraska. “That’s where we feel that our system can make key breakthroughs, as it can quickly learn for the given data and workload what query plans to use and which ones to avoid.”

Kraska says that in contrast to other learning-based approaches to query optimization, Bao learns much faster and can outperform open-source and commercial optimizers with as little as one hour of training time.In the future, his team aims to integrate Bao into cloud systems to improve resource utilization in environments where disk, RAM, and CPU time are scarce resources.

“Our hope is that a system like this will enable much faster query times, and that people will be able to answer questions they hadn’t been able to answer before,” says Kraska.

A related paper about Tsunami was co-written by Kraska, PhD students Jialin Ding and Vikram Nathan, and MIT Professor Mohammad Alizadeh. A paper about Bao was co-written by Kraska, Marcus, PhD students Parimarjan Negi and Hongzi Mao, visiting scientist Nesime Tatbul, and Alizadeh.

The work was done as part of the Data System and AI Lab (DSAIL@CSAIL), which is sponsored by Intel, Google, Microsoft, and the U.S. National Science Foundation. 

Covid Abruzzo Basilicata Calabria Campania Emilia Romagna Friuli Venezia Giulia Lazio Liguria Lombardia Marche Molise Piemonte Puglia Sardegna Sicilia Toscana Trentino Alto Adige Umbria Valle d’Aosta Veneto Italia Agrigento Alessandria Ancona Aosta Arezzo Ascoli Piceno Asti Avellino Bari Barletta-Andria-Trani Belluno Benevento Bergamo Biella Bologna Bolzano Brescia Brindisi Cagliari Caltanissetta Campobasso Carbonia-Iglesias Caserta Catania Catanzaro Chieti Como Cosenza Cremona Crotone Cuneo Enna Fermo Ferrara Firenze Foggia Forlì-Cesena Frosinone Genova Gorizia Grosseto Imperia Isernia La Spezia L’Aquila Latina Lecce Lecco Livorno Lodi Lucca Macerata Mantova Massa-Carrara Matera Messina Milano Modena Monza e della Brianza Napoli Novara Nuoro Olbia-Tempio Oristano Padova Palermo Parma Pavia Perugia Pesaro e Urbino Pescara Piacenza Pisa Pistoia Pordenone Potenza Prato Ragusa Ravenna Reggio Calabria Reggio Emilia Rieti Rimini Roma Rovigo Salerno Medio Campidano Sassari Savona Siena Siracusa Sondrio Taranto Teramo Terni Torino Ogliastra Trapani Trento Treviso Trieste Udine Varese Venezia Verbano-Cusio-Ossola Vercelli Verona Vibo Valentia Vicenza Viterbo

Real-time-data-for-a-better-response-to-disease-outbreaks.jpg

Real-time data for a better response to disease outbreaks | MIT News

news .mit.edu/sites/default/files/styles/news_article__cover_image__original/public/images/202008/MIT-Kinsa-Health-01.jpg?itok=2-Ewyd-3″ />

Kinsa was founded by MIT alumnus Inder Singh MBA ’06, SM ’07 in 2012, with the mission of collecting information about when and where infectious diseases are spreading in real-time. Today the company is fulfilling that mission along several fronts.

It starts with families. More than 1.5 million of Kinsa’s “smart” thermometers have been sold or given away across the country, including hundreds of thousands to families from low-income school districts. The thermometers link to an app that helps users decide if they should seek medical attention based on age, fever, and symptoms.

At the community level, the data generated by the thermometers are anonymized and aggregated, and can be shared with parents and school officials, helping them understand what illnesses are going around and prevent the spread of disease in classrooms.

By working with over 2,000 schools to date in addition to many businesses, Kinsa has also developed predictive models that can forecast flu seasons each year. In the spring of this year, the company showed it could predict flu spread 12-20 weeks in advance at the city level.

The milestone prepared Kinsa for its most profound scale-up yet. When Covid-19 came to the U.S., the company was able to estimate its spread in real-time by tracking fever levels above what would normally be expected. Now Kinsa is working with health officials in five states and three cities to help contain and control the virus.

“By the time the CDC [U.S. Centers for Disease Control] gets the data, it has been processed, deidentified, and people have entered the health system to see a doctor,” say Singh, who is Kinsa’s CEO as well as its founder. “There’s a huge delay from when someone contracts an illness and when they see a doctor. The current health care system only sees the latter; we see the former.”

Today Kinsa finds itself playing a central role in America’s Covid-19 response. In addition to its local partnerships, the company has become a central information hub for the public, media, and researchers with its Healthweather tool, which maps unusual rates of fevers — among the most common symptom of Covid-19 — to help visualize the prevalence of illness in communities.

Singh says Kinsa’s data complement other methods of containing the virus like testing, contact tracing, and the use of face masks.

Better data for better responses

Singh’s first exposure to MIT came while he was attending the Harvard University Kennedy School of Government as a graduate student.

“I remember I interacted with some MIT undergrads, we brainstormed some social-impact ideas,” Singh recalls. “A week later I got an email from them saying they’d prototyped what we were talking about. I was like, ‘You prototyped what we talked about in a week!?’ I was blown away, and it was an insight into how MIT is such a do-er campus. It was so entrepreneurial. I was like, ‘I want to do that.’”

Soon Singh enrolled in the Harvard-MIT Program in Health Sciences and Technology, an interdisciplinary program where Singh earned his master’s and MBA degrees while working with leading research hospitals in the area. The program also set him on a course to improve the way we respond to infectious disease.

Following his graduation, he joined the Clinton Health Access Initiative (CHAI), where he brokered deals between pharmaceutical companies and low-resource countries to lower the cost of medicines for HIV, malaria, and tuberculosis. Singh described CHAI as a dream job, but it opened his eyes to several shortcomings in the global health system.

“The world tries to curb the spread of infectious illness with almost zero real-time information about when and where disease is spreading,” Singh says. “The question I posed to start Kinsa was ‘how do you stop the next outbreak before it becomes an epidemic if you don’t know where and when it’s starting and how fast it’s spreading’?”

Kinsa was started in 2012 with the insight that better data were needed to control infectious diseases. In order to get that data, the company needed a new way of providing value to sick people and families.

“The behavior in the home when someone gets sick is to grab the thermometer,” Singh says. “We piggy-backed off of that to create a communication channel to the sick, to help them get better faster.”

Kinsa started by selling its thermometers and creating a sponsorship program for corporate donors to fund thermometer donations to Title 1 schools, which serve high numbers of economically disadvantaged students. Singh says 40 percent of families that receive a Kinsa thermometer through that program did not previously have any thermometer in their house.

The company says its program has been shown to help schools improve attendance, and has yielded years of real-time data on fever rates to help compare to official estimates and develop its models.

“We had been forecasting flu incidence accurately several weeks out for years, and right around early 2020, we had a massive breakthrough,” Singh recalls. “We showed we could predict flu 12 to 20 weeks out — then March hit. We said, let’s try to remove the fever levels associated with cold and flu from our observed real time signal. What’s left over is unusual fevers, and we saw hotspots across the country. We observed six years of data and there’d been hot spots, but nothing like we were seeing in early March.”

The company quickly made their real-time data available to the public, and on March 14, Singh got on a call with the former New York State health commissioner, the former head of the U.S. Food and Drug Administration, and the man responsible for Taiwan’s successful Covid-19 response.

“I said, ‘There’s hotspots everywhere,” Singh recalls. “They’re in New York, around the Northeast, Texas, Michigan. They said, ‘This is interesting, but it doesn’t look credible because we’re not seeing case reports of Covid-19.’ Low and behold, days and weeks later, we saw the Covid cases start building up.”

A tool against Covid-19

Singh says Kinsa’s data provide an unprecedented look into the way a disease is spreading through a community.

“We can predict the entire incidence curve [of flu season] on a city-by-city basis,” Singh says. “The next best model is [about] three weeks out, at a multistate level. It’s not because we’re smarter than others; it’s because we have better data. We found a way to communicate with someone consistently when they’ve just fallen ill.”

Kinsa has been working with health departments and research groups around the country to help them interpret the company’s data and react to early warnings of Covid-19’s spread. It’s also helping companies around the country as they begin bringing employees back to offices.

Now Kinsa is working on expanding its international presence to help curb infectious diseases on multiple fronts around the world, just like it’s doing in the U.S. The company’s progress promises to help authorities monitor diseases long after Covid-19.

“I started Kinsa to create a global, real-time outbreak monitoring and detection system, and now we have predictive power beyond that,” Singh says. “When you know where and when symptoms are starting and how fast they’re spreading, you can empower local individuals, families, communities, and governments.”

Covid Abruzzo Basilicata Calabria Campania Emilia Romagna Friuli Venezia Giulia Lazio Liguria Lombardia Marche Molise Piemonte Puglia Sardegna Sicilia Toscana Trentino Alto Adige Umbria Valle d’Aosta Veneto Italia Agrigento Alessandria Ancona Aosta Arezzo Ascoli Piceno Asti Avellino Bari Barletta-Andria-Trani Belluno Benevento Bergamo Biella Bologna Bolzano Brescia Brindisi Cagliari Caltanissetta Campobasso Carbonia-Iglesias Caserta Catania Catanzaro Chieti Como Cosenza Cremona Crotone Cuneo Enna Fermo Ferrara Firenze Foggia Forlì-Cesena Frosinone Genova Gorizia Grosseto Imperia Isernia La Spezia L’Aquila Latina Lecce Lecco Livorno Lodi Lucca Macerata Mantova Massa-Carrara Matera Messina Milano Modena Monza e della Brianza Napoli Novara Nuoro Olbia-Tempio Oristano Padova Palermo Parma Pavia Perugia Pesaro e Urbino Pescara Piacenza Pisa Pistoia Pordenone Potenza Prato Ragusa Ravenna Reggio Calabria Reggio Emilia Rieti Rimini Roma Rovigo Salerno Medio Campidano Sassari Savona Siena Siracusa Sondrio Taranto Teramo Terni Torino Ogliastra Trapani Trento Treviso Trieste Udine Varese Venezia Verbano-Cusio-Ossola Vercelli Verona Vibo Valentia Vicenza Viterbo

1598763608_908_How-to-use-Seaborn-Data-Visualization-for-Machine-Learning.png

How to use Seaborn Data Visualization for Machine Learning

Last Updated on August 19, 2020

Data visualization provides insight into the distribution and relationships between variables in a dataset.

This insight can be helpful in selecting data preparation techniques to apply prior to modeling and the types of algorithms that may be most suited to the data.

Seaborn is a data visualization library for Python that runs on top of the popular Matplotlib data visualization library, although it provides a simple interface and aesthetically better-looking plots.

In this tutorial, you will discover a gentle introduction to Seaborn data visualization for machine learning.

After completing this tutorial, you will know:

  • How to summarize the distribution of variables using bar charts, histograms, and box and whisker plots.
  • How to summarize relationships using line plots and scatter plots.
  • How to compare the distribution and relationships of variables for different class values on the same plot.

Kick-start your project with my new book Machine Learning Mastery With Python, including step-by-step tutorials and the Python source code files for all examples.

Let’s get started.

How to use Seaborn Data Visualization for Machine Learning

How to use Seaborn Data Visualization for Machine Learning
Photo by Martin Pettitt, some rights reserved.

Tutorial Overview

This tutorial is divided into six parts; they are:

  • Seaborn Data Visualization Library
  • Line Plots
  • Bar Chart Plots
  • Histogram Plots
  • Box and Whisker Plots
  • Scatter Plots

Seaborn Data Visualization Library

The primary plotting library for Python is called Matplotlib.

Seaborn is a plotting library that offers a simpler interface, sensible defaults for plots needed for machine learning, and most importantly, the plots are aesthetically better looking than those in Matplotlib.

Seaborn requires that Matplotlib is installed first.

You can install Matplotlib directly using pip, as follows:


Once installed, you can confirm that the library can be loaded and used by printing the version number, as follows:


Running the example prints the current version of the Matplotlib library.


Next, the Seaborn library can be installed, also using pip:


Once installed, we can also confirm the library can be loaded and used by printing the version number, as follows:


Running the example prints the current version of the Seaborn library.


To create Seaborn plots, you must import the Seaborn library and call functions to create the plots.

Importantly, Seaborn plotting functions expect data to be provided as Pandas DataFrames. This means that if you are loading your data from CSV files, you must use Pandas functions like read_csv() to load your data as a DataFrame. When plotting, columns can then be specified via the DataFrame name or column index.

To show the plot, you can call the show() function on Matplotlib library.


Alternatively, the plots can be saved to file, such as a PNG formatted image file. The savefig() Matplotlib function can be used to save images.


Now that we have Seaborn installed, let’s look at some common plots we may need when working with machine learning data.

Line Plots

A line plot is generally used to present observations collected at regular intervals.

The x-axis represents the regular interval, such as time. The y-axis shows the observations, ordered by the x-axis and connected by a line.

A line plot can be created in Seaborn by calling the lineplot() function and passing the x-axis data for the regular interval, and y-axis for the observations.

We can demonstrate a line plot using a time series dataset of monthly car sales.

The dataset has two columns: “Month” and “Sales.” Month will be used as the x-axis and Sales will be plotted on the y-axis.


Tying this together, the complete example is listed below.


Running the example first loads the time series dataset and creates a line plot of the data, clearly showing a trend and seasonality in the sales data.

Line Plot of a Time Series Dataset

Line Plot of a Time Series Dataset

For more great examples of line plots with Seaborn, see: Visualizing statistical relationships.

Bar Chart Plots

A bar chart is generally used to present relative quantities for multiple categories.

The x-axis represents the categories that are spaced evenly. The y-axis represents the quantity for each category and is drawn as a bar from the baseline to the appropriate level on the y-axis.

A bar chart can be created in Seaborn by calling the countplot() function and passing the data.

We will demonstrate a bar chart with a variable from the breast cancer classification dataset that is comprised of categorical input variables.

We will just plot one variable, in this case, the first variable which is the age bracket.


Tying this together, the complete example is listed below.


Running the example first loads the breast cancer dataset and creates a bar chart plot of the data, showing each age group and the number of individuals (samples) that fall within reach group.

Bar Chart Plot of Age Range Categorical Variable

Bar Chart Plot of Age Range Categorical Variable

We might also want to plot the counts for each category for a variable, such as the first variable, against the class label.

This can be achieved using the countplot() function and specifying the class variable (column index 9) via the “hue” argument, as follows:


Tying this together, the complete example is listed below.


Running the example first loads the breast cancer dataset and creates a bar chart plot of the data, showing each age group and the number of individuals (samples) that fall within each group separated by the two class labels for the dataset.

Bar Chart Plot of Age Range Categorical Variable by Class Label

Bar Chart Plot of Age Range Categorical Variable by Class Label

For more great examples of bar chart plots with Seaborn, see: Plotting with categorical data.

Histogram Plots

A histogram plot is generally used to summarize the distribution of a numerical data sample.

The x-axis represents discrete bins or intervals for the observations. For example, observations with values between 1 and 10 may be split into five bins, the values [1,2] would be allocated to the first bin, [3,4] would be allocated to the second bin, and so on.

The y-axis represents the frequency or count of the number of observations in the dataset that belong to each bin.

Essentially, a data sample is transformed into a bar chart where each category on the x-axis represents an interval of observation values.

A histogram can be created in Seaborn by calling the distplot() function and passing the variable.

We will demonstrate a boxplot with a numerical variable from the diabetes classification dataset. We will just plot one variable, in this case, the first variable, which is the number of times that a patient was pregnant.


Tying this together, the complete example is listed below.


Running the example first loads the diabetes dataset and creates a histogram plot of the variable, showing the distribution of the values with a hard cut-off at zero.

The plot shows both the histogram (counts of bins) as well as a smooth estimate of the probability density function.

Histogram Plot of Number of Times Pregnant Numerical Variable

Histogram Plot of Number of Times Pregnant Numerical Variable

For more great examples of histogram plots with Seaborn, see: Visualizing the distribution of a dataset.

Box and Whisker Plots

A box and whisker plot, or boxplot for short, is generally used to summarize the distribution of a data sample.

The x-axis is used to represent the data sample, where multiple boxplots can be drawn side by side on the x-axis if desired.

The y-axis represents the observation values. A box is drawn to summarize the middle 50 percent of the dataset starting at the observation at the 25th percentile and ending at the 75th percentile. This is called the interquartile range, or IQR. The median, or 50th percentile, is drawn with a line.

Lines called whiskers are drawn extending from both ends of the box, calculated as (1.5 * IQR) to demonstrate the expected range of sensible values in the distribution. Observations outside the whiskers might be outliers and are drawn with small circles.

A boxplot can be created in Seaborn by calling the boxplot() function and passing the data.

We will demonstrate a boxplot with a numerical variable from the diabetes classification dataset. We will just plot one variable, in this case, the first variable, which is the number of times that a patient was pregnant.


Tying this together, the complete example is listed below.


Running the example first loads the diabetes dataset and creates a boxplot plot of the first input variable, showing the distribution of the number of times patients were pregnant.

We can see the median just above 2.5 times, some outliers up around 15 times (wow!).

Box and Whisker Plot of Number of Times Pregnant Numerical Variable

Box and Whisker Plot of Number of Times Pregnant Numerical Variable

We might also want to plot the distribution of the numerical variable for each value of a categorical variable, such as the first variable, against the class label.

This can be achieved by calling the boxplot() function and passing the class variable as the x-axis and the numerical variable as the y-axis.


Tying this together, the complete example is listed below.


Running the example first loads the diabetes dataset and creates a boxplot of the data, showing the distribution of the number of times pregnant as a numerical variable for the two-class labels.

Box and Whisker Plot of Number of Times Pregnant Numerical Variable by Class Label

Box and Whisker Plot of Number of Times Pregnant Numerical Variable by Class Label

Scatter Plots

A scatter plot, or scatterplot, is generally used to summarize the relationship between two paired data samples.

Paired data samples mean that two measures were recorded for a given observation, such as the weight and height of a person.

The x-axis represents observation values for the first sample, and the y-axis represents the observation values for the second sample. Each point on the plot represents a single observation.

A scatterplot can be created in Seaborn by calling the scatterplot() function and passing the two numerical variables.

We will demonstrate a scatterplot with two numerical variables from the diabetes classification dataset. We will plot the first versus the second variable, in this case, the first variable, which is the number of times that a patient was pregnant, and the second is the plasma glucose concentration after a two hour oral glucose tolerance test (more details of the variables here).


Tying this together, the complete example is listed below.


Running the example first loads the diabetes dataset and creates a scatter plot of the first two input variables.

We can see a somewhat uniform relationship between the two variables.

Scatter Plot of Number of Times Pregnant vs. Plasma Glucose Numerical Variables

Scatter Plot of Number of Times Pregnant vs. Plasma Glucose Numerical Variables

We might also want to plot the relationship for the pair of numerical variables against the class label.

This can be achieved using the scatterplot() function and specifying the class variable (column index 8) via the “hue” argument, as follows:


Tying this together, the complete example is listed below.


Running the example first loads the diabetes dataset and creates a scatter plot of the first two variables vs. class label.

Scatter Plot of Number of Times Pregnant vs. Plasma Glucose Numerical Variables by Class Label

Scatter Plot of Number of Times Pregnant vs. Plasma Glucose Numerical Variables by Class Label

Further Reading

This section provides more resources on the topic if you are looking to go deeper.

Tutorials
APIs

Summary

In this tutorial, you discovered a gentle introduction to Seaborn data visualization for machine learning.

Specifically, you learned:

  • How to summarize the distribution of variables using bar charts, histograms, and box and whisker plots.
  • How to summarize relationships using line plots and scatter plots.
  • How to compare the distribution and relationships of variables for different class values on the same plot.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Discover Fast Machine Learning in Python!

Master Machine Learning With Python
Develop Your Own Models in Minutes

…with just a few lines of scikit-learn code

Learn how in my new Ebook:
Machine Learning Mastery With Python

Covers self-study tutorials and end-to-end projects like:
Loading data, visualization, modeling, tuning, and much more…

Finally Bring Machine Learning To

Your Own Projects

Skip the Academics. Just Results.

See What’s Inside

Covid Abruzzo Basilicata Calabria Campania Emilia Romagna Friuli Venezia Giulia Lazio Liguria Lombardia Marche Molise Piemonte Puglia Sardegna Sicilia Toscana Trentino Alto Adige Umbria Valle d’Aosta Veneto Italia Agrigento Alessandria Ancona Aosta Arezzo Ascoli Piceno Asti Avellino Bari Barletta-Andria-Trani Belluno Benevento Bergamo Biella Bologna Bolzano Brescia Brindisi Cagliari Caltanissetta Campobasso Carbonia-Iglesias Caserta Catania Catanzaro Chieti Como Cosenza Cremona Crotone Cuneo Enna Fermo Ferrara Firenze Foggia Forlì-Cesena Frosinone Genova Gorizia Grosseto Imperia Isernia La Spezia L’Aquila Latina Lecce Lecco Livorno Lodi Lucca Macerata Mantova Massa-Carrara Matera Messina Milano Modena Monza e della Brianza Napoli Novara Nuoro Olbia-Tempio Oristano Padova Palermo Parma Pavia Perugia Pesaro e Urbino Pescara Piacenza Pisa Pistoia Pordenone Potenza Prato Ragusa Ravenna Reggio Calabria Reggio Emilia Rieti Rimini Roma Rovigo Salerno Medio Campidano Sassari Savona Siena Siracusa Sondrio Taranto Teramo Terni Torino Ogliastra Trapani Trento Treviso Trieste Udine Varese Venezia Verbano-Cusio-Ossola Vercelli Verona Vibo Valentia Vicenza Viterbo

Recent Posts

Archives