Pilot projects to drive collaborative working

Collaborative working is a founding principle of ECMWF and one of our key strengths. The concept of ‘pilot projects’ was established by our Council in December 2022 as a route to enhance collaboration on specific aspects of our work and share expertise with our Member and Co-operating States.

Making effective operational use of non-standard IoT observing systems is hugely challenging. But we believe the potential benefits are equally significant and that the key to unlocking this potential lies in the development of autonomous machine-learned assimilation systems.

Tony McNally, Head of Earth System Assimilation, ECMWF

Through our pilot project on adapting to emerging technologies, we are paving the way for national weather services to deploy data services using the polytope/FDB software stack, allowing efficient feature extraction from large hypercubes of operational numerical weather prediction data.

Carlos Osuna, Lead computing team in numerical development, MeteoSwiss

It is important that the benefits of any selected pilot projects should link to our Strategy and focus on areas where activities can be challenging to carry out in isolation. This is of particular value in activities where expertise and excellence within the Member States can complement our core competences. The overall aim is to provide benefits both to ECMWF and to all Member and Co-operating States.

Two pilot projects were initially agreed by the Council in December 2022: ‘Internet of Things (IoT) observations for numerical weather prediction’ and ‘Adaptation to emerging technologies’.

Many countries expressed their interest in these pilots, so in March 2023, ECMWF organised two workshops to refine the scope of the projects and select lead organisations. An important aspect of the projects is the organisation of workshops and other knowledge dissemination events. By the end of the year, both projects were set up and ready to start.

The IoT pilot project, led by the EUMETNET network of 33 European national meteorological services, is an initial exploration of how data from non-traditional sources might be exploited. The focus includes data from private weather stations and smartphone pressure observations.

The project will first explore how such data are currently collected, managed, quality controlled and distributed. The quality of the data must be consistently high, which can be a challenge with new types of observations. If those issues can be successfully addressed, being able to tap into potentially vast amounts of data, particularly where more traditional observations are less available, could have huge benefits for weather forecasting and for verification. The project aims to build a test community-based platform to ingest, pre-process, format and make available the data.

Making effective operational use of non-standard IoT observing systems is hugely challenging. But we believe the potential benefits are equally significant and that the key to unlocking this potential lies in the development of autonomous machine-learned assimilation systems.

Tony McNally, Head of Earth System Assimilation, ECMWF

Through our pilot project on adapting to emerging technologies, we are paving the way for national weather services to deploy data services using the polytope/FDB software stack, allowing efficient feature extraction from large hypercubes of operational numerical weather prediction data.

Carlos Osuna, Lead computing team in numerical development, MeteoSwiss

The second project, led by MeteoSwiss, will look at adapting existing code to support Member States in making use of emerging technologies. This would allow them to modify their workflows to maximise opportunities arising from advances in high-performance computing processing and storage. It will explore the use of such technologies for data production and data sharing, while developing workflows that involve our supercomputing and European Weather Cloud resources. There will also be regular dialogue on scientific supercomputing and on merging adaptation strategies. Part of the project will be to explore funding opportunities to sustain these activities in the long term.

The project is being developed in parallel with the EU’s Destination Earth initiative to create digital twins of the Earth system. Solutions and technologies built as a result of the research into developing Destination Earth will, through this project, become available to all our Member States.

At the end of 2023, the Council approved a third pilot project, on artificial intelligence and machine learning, recognised as a disruptive technology that requires new ways of collaborating. We will work with experts from the national meteorological services of Member and Co-operating States to co-develop and evaluate a range of machine learning approaches to global, regional and local modelling, with an emphasis on open development and knowledge sharing. The lead coordinators are Met Norway and MeteoSwiss, and the aim is to ensure that all the entities of the European Meteorological Infrastructure maximise the benefits from the activities in this rapidly evolving domain. There is particular emphasis on creating a supporting infrastructure and taking the machine learning developments into operations, along with the necessary training and support.

Background image:© monsitj/iStock/Getty Images Plus