Grids: The Next Generation

The topic of Grids was last addressed in Issue 45 of ERCIM News, in April 2001. At that time the field was relatively new. In the last few years, however, significant efforts have been made all over Europe to establish production Grids, in order to research every aspect and potential application of Grid systems. The goal of the current thematic issue is to show both selected results from this research and applications of Grid systems, and to show the goals of some newly launched national and EU projects.

The issue contains 32 papers that can be divided into the following main areas:

• Infrastructure
• Architecture
• Middleware
• Programming
• Applications
• New projects

For rigorous research in this field, it is necessary for Grid infrastructure to be developed. Unfortunately, few functional Grid infrastructures have so far been created in Europe. It is therefore encouraging to find four papers reporting on working Grid infrastructures, and three others detailing plans for the creation of infrastructures. The largest production Grid infrastructure in Europe is the LCG Grid developed by CERN and other physics institutes in collaboration with the EDG (European DataGrid) project. This was successfully completed in March 2004.

The first paper describes in detail the UK Computing Grid and its relationship with LCG. LCG is the largest functioning grid in the world, with over 5000 CPUs and almost 4000 TB of storage at more than 70 sites around the world. The second paper reports on the CERN openlab project, a collaboration between CERN and industrial partners to further develop the LCG Grid infrastructure using the latest industrial technologies. While the LCG’s middleware had initially been targeted solely for the x86 platform, the current goal is to port the middleware to Itanium. Another major focus of the project is the ‘10 Gigabit challenge’, ie how to achieve data transfer rates of 10 Gb/s both within the opencluster and to other sites. Other important issues include virtualization and automated installation of grid nodes. The Fraunhofer Resource Grid has been running since 2002 in Germany and combines the computing and storage infrastructure of the six institutes of the Fraunhofer Grid Alliance. This infrastructure has been successfully applied to solving engineering problems such as casting process optimization, microstructure simulation and drug design. The Hungarian ClusterGrid is a production grid that has been operating since mid-2002, and currently involves 1100 nodes and provides 500 Gflops of supercomputing power for Hungarian academic users. Its novelty is that the component clusters serve educational purposes during the day and are connected to the ClusterGrid during nights and weekends. The result is a Grid solution with the extremely inexpensive annual operation cost of 40,000 Euro.

Icatis has developed technology similar to the Hungarian ClusterGrid: during cluster usage peaks, a number of idle user machines can be aggregated to the cluster. The system is designed to be transparent to the PC owners, and the machines are only used at times when they are likely to be idle (nights, weekends, vacations etc).

Two special-purpose Grid infrastructures are under preparation and are reported in this issue. The first is the European Learning Grid infrastructure, which integrates service-oriented Grid technology with semantic technology for the creation of future learning scenarios. The second is the DILIGENT project, which will result in a digital library infrastructure based on Grid-enabled technology. This will allow members of dynamic virtual organizations to collaborate by exploiting shared knowledge and physical resources.

Grid architecture is another exciting subject and is addressed by four papers in this issue. One of the most traditional types of Grid architecture is the cycle-harvesting grid. Work at IBM Research has focused on the performance issues of such architectures. Their performance-related work led to a definition of quality of service known as Hard Statistical QoS (HSQ) and an accompanying system architecture that can support commercial service contracts. The JGrid project is an interesting and novel approach to building service-oriented Grid architectures based on the Jini technology and Java platform of Sun Microsystems. The JGrid project extends the look-up service of Jini with a generic Grid discovery subsystem and with a compute broker. Interactive computing tasks are executed by compute services, jobs are executed by batch services, and storage services allow users to store and retrieve their data from anywhere in the world. A major obstacle to applying Grid technology in industry is the lack of a Grid-based ASP model. The GRASP project is an industry-driven European research project exploring the use of the Grid services concept for solving this problem. To achieve this, GRASP has developed an architectural framework for Grid-based Application Service Provision and a prototype realization of this framework. Since mobile communication technology is widely used in everyday life, its integration with Grid technology will radically advance the pervasiveness of Grid computing across Europe. Through the integration of these two technologies, the Akogrimo project intends to create a commercial operator-oriented architecture and platform that support the deployment of Grid services worldwide.

Grid Middleware
Grid middleware design is a central focus of Grid research. Six papers in this issue address this extremely important subject. The first introduces a new component-based middleware approach designed as the next generation of Grid middleware. The new concept will allow automatic deployment of software components on large-scale distributed systems and grids while making the best use of available resources. The project involves the design of a runtime system, called PadicoTM, to allow components to exchange data independently of the underlying networking technologies. The appropriate data access and integration support in the middleware is an important challenge. The UK’s OGSA-DAI system provides a framework and set of components operating across grids or Web services that deliver these mechanisms reliably. Using this framework, many projects are building an OGSA-DAI and extending its repertoire. It already handles a range of relational systems, XML databases and collections of files.

Unsolved security issues such as authentication and authorization remain major obstacles to the industrial implementation of grids. The research area of Grid accounting and economy systems has also been neglected. The NIKHEF and UvA teams are jointly developing a combined authentication, authorization and accounting (AAA) service technique. Based on this, site-local authorization and creation of virtual organization will become more dynamic and easier for both users and administrators. A particular advantage of this approach is that rather than starting everything from scratch, they are building on the Virtual Organization Management Service (VOMS) developed by INFN in Italy. This is one of the rare cases in which one project fertilizes another. Unfortunately not many papers report such result transfer between projects.

Resource management is a classic and central issue in Grid middleware design and it is therefore unsurprising that two papers deal with this area of research. Within the framework of the German VIOLA project, new Grid middleware components will be developed to exploit the capabilities of the new underlying optical network infrastructure. These new components will include a MetaScheduler (allowing co-allocation of compute resources, network resources with necessary QoS or other resources like visualization devices) and MetaMPICH communication libraries for distributed execution of parallel applications using MPI for inter-process communication. The GridCat group at UPC is working on integrating Grid technology and software-agent technology in order to create an agent-based Grid Resource Management system (GRM). Grids and agent communities offer two approaches to open distributed systems. Intelligent cooperation, coordination and negotiation are issues handled by software agents and are required by Grid users and Grid resource providers in order to efficiently inter-operate. Another project working on the merging of Grid and agent technologies addresses QoS problems. In their proposed architecture, self-interested agents will interact using electronic market-based techniques with the goal of establishing and maintaining a certain level of QoS in the Grid network.

High-Level Grid Programming Concepts
The next field covered by this issue is frequently neglected. Since there is a certain gap between the end-user needs and the access level provided by Grid middleware, in many cases end-users must learn low-level command interfaces if they wish to use the Grid. Moreover they must be fully aware of the underlying Grid middleware concepts and re-learn the concepts and access mechanisms every time a new generation of Grid middleware is introduced.

High-level Grid programming concepts and supporting tools and portals are therefore extremely important for end-users, making access to Grid services more convenient. Four papers of the current issue describe concepts, tools and portals that will make the lives of end-users much easier. The Spanish GRID-IT project introduced a new middleware design that offers an object-oriented, high-level application programming interface, which simplifies the process of remote task execution in Grid deployment. It is intended to give support for parameter sweep applications. The user is provided with a high-level API that exposes a very natural and convenient point of entry to the Grid services.

The well-known workflow concept facilitates Grid programming, and is handled by two papers. Both of these emphasize the potential commercial/business applications of grids based on their workflow concept. The first paper extends the capability of workflow systems in Grid environments, generally focusing on the capacity to enable more reliable, trustworthy and efficient use of Grid resources. One of the biggest obstacles to widespread industrial utilization of Grid technology is the existence of large numbers of applications written in legacy code, which are inaccessible as Grid services. The second paper in this area introduces a new approach known as GEMLCA (Grid Execution Management for Legacy Code Architecture), to deploy legacy codes as Grid services without modifying the original code. Moreover, such legacy code services can easily be applied by the end-user within the P-GRADE portal, providing visual workflow construction facilities. With simple drag-and-drop style programming, users can build complex workflows from GEMLCA legacy code services. These can be run on complex Grid systems such as GT-3 grids, while the low-level details of that grid remain completely hidden to the user.

Another visual service composition system is Jopera, consisting of a visual composition language and a set of integrated software development tools for composing Grid services. Jopera offers an open, flexible platform for Grid service composition that significantly facilitates the work of end-users in accessing the Grid.

The final and perhaps most important class of papers demonstrates that the Grid really is able to contribute to the realization of e-science. They also show different application areas in which Grid technology can provide significant support for the fast and efficient solution of large scientific problems. The particle physics application has already been mentioned in the context of existing production Grid infrastructures, but other application areas are also covered.

The first paper in this class describes work on Grid-enabled Weka, a widely used toolkit for machine learning and data mining. It is a large collection of state-of-the-art machine-learning algorithms written in Java. Weka contains tools for classification, regression clustering, association rules, visualization, and data preprocessing. In Grid-enabled Weka, execution of these tasks can be distributed across several computers in an ad hoc Grid. The second describes the eMinerals project, one of the NERC e-science testbed projects. The scientific aim is to use Grid technology to advance the capabilities of molecular-scale simulations for the study of environmental processes. The third application area is extremely important and will open new horizons for medical treatments. Medical simulations and visualizations typically require computational power at a level usually unavailable in a hospital. The University of Amsterdam recently demonstrated Virtual Vascular Surgery (a virtual bypass operation), in which large-scale simulation and visualization capabilities were offered as services on a grid. Visualization can be further improved and made photorealistic through the use of Grid technology. The fourth paper reports that at VTT, Grid-based distributed computing was utilized in order to produce photorealistic images in a faster and cheaper way. A new grid known as Grix was developed based on platform-neutral architecture.

Complex problems that can only be solved in non-polynomial time are becoming common in many domains of our lives: economy, industrial environments, bioinformatics, etc. For such a wide spectrum of problems, heuristics come to the rescue, since exact techniques are unable to locate any kind of solution. The fifth paper addresses these issues, showing current research on solving optimization problems with Grid-enabled technology like Globus, Condor, Legion and SGE. Bioinformatics has already been mentioned but the last paper in this group puts it in a new context. Fully annotated data Grid services have enormous relevance in bioinformatics, especially systems biology, due to the vast number of biochemical entities and interactions that need to be kept track of. In order to tackle the problem, an annotation tool for data in Grid services has been developed by the Bioinformatics Department and the Department for Web Applications of Fraunhofer Institute.

Such a theme as this is incomplete without a look at plans for creating the next generation of grids. Four papers are reporting newly launched projects that address a variety of aspects of future research. The CoreGRID Network of Excellence project is probably the largest and most influential of these in Europe. It commenced activity on 1st September 2004 with the aim of allowing European researchers to develop next-generation Grid middleware through a high-level joint program of activities. Another important European project is GridCoord, which is a Special Support Action project. The goal is the coordination of European initiatives and research activities in Grid computing in order to strengthen cooperation between agencies planning future activities, to enhance collaboration between research and user communities, and to develop visionary national and EU programs and roadmaps.

Yet another EU project is TrustCom, an integrated project that aims to develop a framework for trust, security and contract management in dynamic virtual organizations. The framework will enable collaborative business processes to be undertaken securely in self-managed and dynamic value-chains of businesses and governments. A fourth paper describes a national Grid project from Italy. Grid.it has a strong interdisciplinary character and aims to define, implement and apply innovative solutions for networked computing-enabling platforms, which are oriented towards scalable virtual organizations.

Finally, we provide a forward-looking perspective with a short paper on Next Generation Grids, which describes the work of the Expert Group convened by the EC and also provides the URLs to the original reports of the group. Considering the importance of this subject we positioned this paper as the first article of this Special Theme section.

Naturally, one thematic issue cannot cover the entire spectrum of current and planned Grid research in Europe. However, we believe that the 32 papers briefly introduced above give a good overview of the major Grid research directions in Europe. From these papers, readers will gain an overall picture of European Grid research, and can position their own strategies on dealing with Grid systems in the future.

Please contact:
Peter Kacsuk, STZAKI, Hungary
E-mail: kacsuklutra.sztaki.hu

Keith Jeffery, CCLRC, UK
E-mail: K.G.Jefferyrl.ac.uk