Data Management in Climate Research
by Kerstin Kleese
Parallel environmental models are currently some of the most demanding codes we have. They push the machines available to their limit and are still in need of more resources. However the bottleneck in running these codes is not so much the code performance as the data handling strategies employed. The High Performance Computing Initiative Centre at CLRC, Daresbury Laboratory (UK) is setting up a new project to investigate this important issue. Besides analysing existing solutions, the project will also look for new, more flexible and portable approaches.
The UK environmental research community relies heavily on High Performance Computing (HPC) to facilitate its studies. Fortunately the environmental model codes have a large potential for parallelization, which has been well explored by numerous research groups in the UK (eg UGAMP and OCCAM). Still increased resolutions, longer runs or larger ensembles have a significant impact on the HPC resources required and can easily overwhelm current systems. Often data handling has the most dramatic influence on model run times or determine whether a model can be run at all. Thus optimal data handling is vital for this type of application. Unfortunately the problems do not stop with the end of a successful model run. These codes produce vast amounts of data which have to be archived for future analyses. Current data storage systems leave something to be desired in speed and ease-of-use. The situation for data retrieval is even worse. Tedious searching for archived data, long waiting times and no selective extraction possibilities are common problems for modern scientists. Sometimes it is faster to run a model again instead of retrieving data from a previous run.
It is already clear that the data requirements of the community will increase even more over the coming years. Big centres like the European Centre for Medium Range Weather Forecast expect the volume of their data archive to double every 18 months. New machine architectures allow potentially larger models to be run. Data handling presents a severe bottleneck for today's science, without new strategies it might prevent future progress.
First steps
For our project it was decided to take a holistic approach, identifying four main areas of interest:
- data handling within the model codes
- file access during run time (on different platforms)
- data archival
- data retrieval.
Although all these areas have been investigated separately, and some interesting in-house solutions exist, little has been done to offer a portable solution that can be easily adapted to the actual requirements of different sites.
The first step is to analyse the current situation. Research results of leading scientific groups concerning data handling strategies within model codes have been examined. A lot of work has been done in this area over the past years, and we can certainly benefit from that. We would like to compare the different approaches, trying to find similarities, differences and tendencies that could be useful for the community. Secondly a list is in preparation covering the different file access mechanisms during run time on various systems. This gives a clear overview about what is available, how fast is it and what the user can do to make the most of it. This information will serve as a base for further investigations. In connection with vendors and other sites we have started to gather more information about the data archival and retrieval systems that are in use today. The clear message so far is that there is a desperate need for more intelligent solutions.
Future
Our project will continue to analyse existing solutions. It will test which data handling strategies within model codes are best for which type of application. We will frequently investigate new machines or relevant changes to existing system architectures. A collaboration with a leading systems house has just started, to determine which off-the-shelf products could be used to provide more flexible data archival and retrieval mechanisms for scientific data.
Please contact:
Kerstin Kleese - CLRC
Tel: +44 1 925 60 3207
E-mail: k.kleese@dl.ac.uk