Data Mining applied to Air Pollution
by Brian J Read
An understanding of the behaviour of air pollution is needed to predict it and then to guide action to ameliorate it. Calculations with dynamical models are based on the relevant physics and chemistry. To help with the design and validation of such models, a complementary approach is described here. It examines data on air quality empirically by Data Mining using, in particular, machine learning techniques, aiming for a better understanding of the phenomenon and a more direct interpretation of the data.
The work of the Database Group at CLRC has long concentrated on the practical application of data management technology. The emphasis is on helping users at the laboratory and externally to exploit the value in data. Implementing databases and providing easy access is the basis for this, supplemented by data exploration and decision support tools. More recently, interest has extended to data mining, or more fully Knowledge Discovery in Databases (KDD). This may be defined as "the non-trivial extraction of implicit, previously unknown and potentially useful knowledge from data". Data mining is just the discovery stage of the whole KDD process. Indeed, most of the work in practice lies in the preparatory stages of data selection and data cleaning. Extensive data exploration is essential if the data mining is to yield intelligent results.
Data mining is multi-disciplinary: it covers expert systems, database technology, statistics, machine learning ("AI") and data visualisation. It goes beyond directed querying of a database (eg by using SQL) by instead looking for hypotheses or questions rather than detailed answers. Most interest is in mining commercial data - for example credit profiling or market basket analysis. However it is starting to be used in scientific applications too. CLRC as a leading research laboratory has masses of data. Thus there is the motivation to see how data mining techniques might supplement the more traditional scientific analysis in formulating and testing hypotheses. Of specific interest are the induction of rules and neural net models. Considering environmental data, measurement and possibly control of air pollution is increasingly topical. In applying the KDD process, our objectives are two-fold:
- to improve our understanding of the relevant factors and their relationships, including the possible discovery of non-obvious features in the data that may suggest better formulations of the physical models
- to induce models solely from the data so that dynamical simulations might be compared to them, and that they may also have utility, offering (short term) predictive power.
The investigation uses urban air quality measurements taken hourly in the City of Cambridge (UK). These are especially useful since simultaneous weather data from the same location are also available. The objectives are, for example, to look for and interpret possible correlations between each pollutant (NO, NO2, NOx, CO, O3 and PM10) and a) the other pollutants b) the weather (wind strength and direction, temperature, relative humidity and radiance) looking in particular for lags that is, one attribute seeming to affect another with a delay of perhaps hours or of days. Other factors are possible. For example, clearly noticeable is lower NOx on Sundays through less traffic.
The initial analysis concentrated on the daily maxima of the pollutants. This simplifies the problem, the results providing a guide for a later full analysis. Also the peak values were further expressed as bands (eg low, medium and high). The bands are directly related to standards or targets recommended by the Expert Panel on Air Quality Standards (EPAQS) that the public can appreciate. The two principal machine learning techniques used are neural networks and the induction of rules and decision trees. Expressing their predictions as band values makes the results of such rules and models easier to understand.
Work so far supports the common experience in data mining that most of the effort is in data preparation and exploration. The data must be cleaned to allow for missing and bad measurements. Detailed examination leads to transforming the data into more effective forms. The modelling process is very iterative, using statistics and visualisation to guide strategy. The temporal dimension with its lagged correlations adds significantly to the search space for the most relevant parameters.
More extensive investigation is needed to establish under what circumstances data mining might be as effective as dynamical modelling. (For instance, urban air quality varies greatly from street to street depending on buildings and traffic.) A feature of data mining is that it can short circuit the post-interpretation of the output of numerical simulations by directly predicting the probability of exceeding pollution thresholds. More generally, data mining analysis might offer a reference model in the validation of simulation calculations.
Please contact:
Brian J Read - CLRC
Tel: +44 1235 44 6492
E-mail: b.j.read@rl.ac.uk