Workflows on Manycore Processors

The project proposal addresses the "systems on chip" specific topic from the "Information and communications technologies (ICT)" thematic area. Nowadays it is widely accepted that, if we are to continue to enjoy doubling the software performance every 18 months, we must change the way in which we design commodity processors by embedding several processing cores operating at lower frequency onto the same processor chip. The consequence of this radical change to the design of applications running on commodity single chip processors is dramatic: we can no longer double their performance by simply doubling the processor clock speed, but we must use parallel computing instead. This development has a profound effect on commercial software development including languages, compilers, operating systems, and tools. Whereas traditional parallel computing systems have been mostly used for running scientific applications on large compute clusters and supercomputers, times have changed with the penetration of manycore processors in all segments of the computing market including desktop systems and servers. Manycore processors have a profound effect on software development in industry, business, and also science, as most applications are not designed to exploit manycore-based parallel computing systems, and the large majority of software developers have little expertise in creating parallel programs. Currently, only a small group of the most highly skilled developers has expertise in parallel programming. Software developers need new programming models, tools, and abstraction by the operating system to handle concurrency and complexity of numerous processors. An important class of applications which has been largely ignored so far when dealing with effective parallelisation for manycore processors is workflow applications. Workflows have a strong impact on application development in industry, commerce and science on desktop, server and parallel computing infrastructures, accelerating and simplifying programming by allowing programmers to focus on the composition of existing legacy programs to create larger and more powerful applications. Workflows have emerged as an easier way to formalise and structure data analysis, to execute the necessary computations on computing resources, to collect information about the derived results and, if necessary, to repeat the analysis. The goal of the project is to propose and investigate workflows as a promising programming paradigm for manycore on-chip processing systems on a chip and to craft a novel development, parallelisation and optimisation environment with the following research contributions: 1. To define BPEL Multi-Processing (BPEL-MP) language for programming manycore processors as an extension to the most widely used language standard for specifying workflow applications (BPEL) covering the most important features for expressing data and task-based parallelism at a high-level of abstraction; 2. To research multi-objective scheduling, enactment, and runtime optimisations techniques to reduce the energy consumption, execution time and computing costs of workflow applications on manycore processors; 3. To provide innovative resource management solutions for manycore architectures, including low energy provisioning methods and economically viable cost models; 4. To make existing BPEL workflows available in the community available on the emerging manycore computing architectures with enhanced performance and reduced cost and consumed energy; 5. To provide a parallelisation environment for workflows that enables new products to be realised with a considerable shorter time-to-market.

For many years, vendors and users got accustomed with the Moores Laws march towards ever-cheaper, ever-faster, and ever-better computing. Today, it is no longer possible to improve the computing performance by increasing the processors clock frequency due the much higher increase in energy consumption. It is widely accepted that, if we are to continue to enjoy doubling the software performance every 18 months, we must embed several processing cores at lower frequency onto the same processing chip and use parallel computing for programming them. This development has a profound effect on commercial software development including languages, compilers, operating systems, and tools. An important class of applications largely ignored so far when dealing with effective parallelization for manycore processors are workflow applications, which have today a strong impact on software development in industry, commerce and science by allowing programmers to focus on the composition of existing legacy programs to create larger and more powerful applications.This project researched and developed new methods that support the execution cycle of workflow applications on manycore parallel architectures through the following innovations:A single interoperable workflow representation, executable by other important workflow systems in the field, enabling interoperability, cooperation, reproducibility and reuse of results among different communities;A complete workflow compilation and runtime execution environment that translates interoperable workflows in stand-alone parallel C++ programs executed on manycore parallel platforms with high-performance and significantly lower overheads compared to other existing interpreted methods;Energy-aware resource management techniques able to accurately simulate and provision resources with reduced energy consumption, which is of extreme importance for next-generation data center infrastructures;Multi-objective scheduling algorithms able to optimize and plan workflow applications on heterogeneous data center infrastructures by modelling and guaranteeing the tradeoff between conflicting parameters such as time, energy, cost, reliability and uncertainty;Real-world inter-disciplinary validation through cooperation with numerous industrial and scientific collaborators from meteorology, theoretical chemistry, astronomy, finances, or mathematics, by optimizing the utilization of their expensive manycore infrastructure while significantly reducing the time to solution of their simulations.