The year saw much progress in science and technology at the Centre. On the scientific front, an upgrade of the Integrated Forecasting System (IFS) to Cycle 48r1 was prepared for implementation in 2023. This will be the first upgrade on the new high-performance computing facility (HPCF) in Bologna, Italy. It will increase the horizontal resolution of our ensemble forecasts from 18 km to 9 km, the current resolution of our single high-resolution forecast.
We started to produce our forecasts on the new Atos HPCF in Bologna in October 2022. As early as March 2022, the Copernicus Climate and Atmosphere Data Stores were successfully migrated from Reading to the Bologna data centre.
Among many other changes, the upgrade will also revise the extended-range configuration from twice-weekly to daily production and from 51 to 101 ensemble members. To help prepare optimal initial conditions of weather forecasts by means of data assimilation, one change in Cycle 48r1 will be the addition of surface-sensitive microwave imager channels over land surfaces.
Research on even finer resolutions in global weather forecasting found that increasing the horizontal resolution of the atmosphere from 9 km in the operational IFS to 4.5 and 1.4 km markedly improves predictions of tropical cyclone intensity. Experimental use of a higher-resolution ocean model revealed for the first time sea ice cracks developing in the Arctic, with repercussions on cloud cover and two-metre temperature.
Monitoring visible radiances by satellites for information on clouds was explored, showing potential benefits for data assimilation. Progress was made in preparing a new record of land cover and vegetation observations for the benefit of reanalysis and seasonal-to-decadal prediction systems. The Ensemble of Data Assimilations was used to show that it would be beneficial to have more passive microwave sounding observations. And progress was made in estimating sea-surface temperature inside ECMWF’s coupled atmosphere–ocean data assimilation system.
We started to produce our forecasts on the new Atos HPCF in Bologna in October 2022. As early as March 2022, the Copernicus Climate and Atmosphere Data Stores were successfully migrated from Reading to the Bologna data centre.
Over the course of 2022, ECMWF was successful in adding 30 externally funded projects to its portfolio. Three of these are projects we coordinate. They will ultimately lead to improvements of the EU-funded Copernicus Climate Change Service (C3S) and Copernicus Atmosphere Monitoring Service (CAMS) we provide.
Scientific advances were supported by progress in the technological field. In particular, we started to produce our forecasts on the new Atos HPCF in Bologna in October 2022. As early as March 2022, the Copernicus Climate and Atmosphere Data Stores (CDS/ADS) run by ECMWF were successfully migrated from Reading to new cloud infrastructure installed in the Bologna data centre.
The IFS was also tested on the world’s most powerful supercomputer in mid-2022, Fugaku, at a global resolution of just 4 km, and ECMWF started to make some of the IFS available on an open-source basis to facilitate collaboration on the code. Finally, we developed a new Software Strategy in 2022.
More developments regarding technology, such as progress in machine learning, preparations for the use of future computing architectures, the move of the Data Handling System to Bologna, and the launch of ECMWF’s participation in the EU’s Destination Earth initiative, are presented in separate sections.
Extended-range forecasts are predictions beyond two weeks but less than a season. In 2022, they were produced at ECMWF as 46-day integrations issued every Monday and Thursday at 00 UTC. They were made up of a 51-member ensemble at a horizontal resolution of about 18 km up to day 15 and about 36 km beyond that.
The availability of the new Atos HPCF in Bologna provided an opportunity to enhance the extended-range forecasting system. From IFS Cycle 48r1, to be introduced in 2023, extended-range forecasts will be produced daily instead of twice weekly, and they will have 101 ensemble members instead of 51. They will be run separately from higher-resolution medium-range forecasts at a horizontal resolution of about 36 km from day 0 to 46.
This will significantly increase the quality of extended-range forecasts for days of the week other than Monday and Thursday. Moreover, doubling the ensemble size to 101 members will provide a more accurate representation of the forecast probability distribution function. In particular, this will ensure a more accurate prediction of the tail of that function, which is important for a more accurate prediction of the probability of extreme events.
Tropical storm strike probability with a 51-member and a 101-member ensemble
Tropical storm strike probability maps for week 4 for the extended-range forecast starting on 7 January 2021, showing (a) the strike probability produced with a 51-member ensemble and (b) the strike probability produced with a 101-member ensemble, both in the real-time forecast configuration. The 101-member ensemble forecast displays higher probabilities of a tropical cyclone strike over the Mozambique Channel and west and northeast of Australia than the 51-member ensemble. It also shows lower probabilities east of 180°E. In the event, tropical cyclones did strike in the first three locations and did not strike in the fourth.
We make extensive use of satellite observations to help establish the initial conditions for our forecasts. We have recently made progress towards an ‘all-surface’ use of satellite microwave observations. Some of these observations will be included in IFS Cycle 48r1 to be introduced in 2023.
A lot of data from surface-sensitive microwave channels has previously been screened out due to surface types that are hard to simulate. These surface types include land, snow, sea ice and mixtures of all surface types. In some channels, the new developments increase the number of observations being assimilated into the forecasting system by around 30%.
One change in Cycle 48r1 will be the addition of surface-sensitive microwave imager channels over land surfaces. This is a first in global weather forecasting. It depends on having a well-developed ‘all-sky’ assimilation framework, in which satellite data are used in areas of cloud and precipitation rather than just in clear-sky areas.
The figure illustrates expected improvements in observational coverage in Cycle 48r1 compared to the current Cycle 47r3. It uses the example of a microwave imager channel from the AMSR2 sensor. The figure shows big additions in coverage over land surfaces. It also reveals that in Cycle 48r1 it is still too difficult to use such observations over sea ice, snow, desert or high-altitude surfaces. However, we expect to see these surface types included in the near future – for example, sea ice is expected to be included in Cycle 49r1.
New observations assimilated
The chart shows the coverage of observations to be assimilated in Cycle 48r1 from the Advanced Microwave Scanning Radiometer – 2 (AMSR2) in channel 11, with black dots indicating data assimilated in Cycle 47r3 and red dots the data added in Cycle 48r1. Data are for the 00 UTC cycle on 20 June 2019.
Simulations showed that increasing the horizontal resolution of the atmosphere from 9 km in the operational IFS to 4.5 and 1.4 km markedly improves predictions of tropical cyclone intensity. It also improves the ocean response to the passing of a tropical cyclone. The simulations were carried out as part of the INCITE 2022 project awarded to ECMWF and US Oak Ridge National Laboratory scientists.
Extreme tropical cyclones Irma (September 2017), Florence (September 2018), Teddy (September 2020) and Ida (August 2021) were chosen for the investigation.
The figure shows that the tropical cyclones are more intense at higher horizontal resolution and in better agreement with observations (Best Track data). This is a result of the fact that tropical cyclones require high spatial resolution to be represented accurately. It was found that the benefit of increased horizontal resolution can already be achieved at 4.5 km.
In addition, it was found that sea-surface temperature during a tropical cyclone is better represented if a higher atmospheric resolution and a higher ocean resolution are used. An accurate representation of sea-surface temperature is important because it may influence the intensity of the tropical cyclone.
Central pressure forecasts for tropical cyclones
The charts show central pressure forecasts up to 6 days ahead for tropical cyclones Irma, Florence, Teddy and Ida at different horizontal resolutions of the atmosphere, and Best Track observations. The ocean resolution is 1/12 degree. The 9 km forecast is currently operational at ECMWF. The higher-resolution forecasts are better able to forecast tropical cyclone intensity.
Such sea-ice-related imprints on the atmosphere have not previously been considered in global Earth system models.
Increasing the horizontal resolution of forecasting models makes it possible to resolve more processes at smaller spatial scales and ultimately improve forecast skill, but conservation properties of models can also have such an impact. As part of the EU-funded Horizon 2020 project nextGEMS, we thus looked into modelling small-scale sea-ice cracks and fixing water and energy budget imbalances.
The operational IFS currently uses the NEMO ocean model at 1/4 degree resolution (about 25 km at the equator). For nextGEMS experiments, the IFS was coupled to the multi-resolution FESOM sea-ice/ocean model. This was run at a resolution of about 4–5 km in mid- and high latitudes.
Such sea-ice-related imprints on the atmosphere have not previously been considered in global Earth system models.
In this high-resolution model of the ocean and sea ice, sea-ice cracks develop in the Arctic. The atmosphere responds to the exposed warm waters within sea-ice cracks through changes in evaporation rates, cloud cover, and two-metre temperature. Such sea-ice-related imprints on the atmosphere have not previously been considered in global Earth system models, such as the IFS.
We also significantly improved the conservation of water and energy in the IFS as part of work that was triggered by the nextGEMS project. The multi-year simulations performed in the project had shown that water and energy imbalances became considerably worse at km-scale resolutions. Ensuring global water conservation now improves the quality even of our medium-range weather forecasts at a resolution of 9 km.
Effects of changes in sea ice on the atmosphere
Sea-ice concentration and two-metre temperature around the New Siberian Islands in the Arctic Ocean in the IFS at a resolution of 2.8 km, coupled to a FESOM sea-ice/ocean model at a resolution of 4 to 5 km in the Arctic. The panels show the results of simulations for 13 February 2020 at 08 UTC.
Visible radiances monitored by satellites contain a wealth of information on clouds. However, they have never been assimilated to help establish the initial conditions of forecasts in global numerical weather prediction models. This is due to the complex scattering of light.
We performed experiments to see whether cloudy visible radiances can be used in the IFS. Due to the complexities of the scattering of visible radiation from land and ice surfaces, only data over ice-free oceans were used.
Data were obtained from the Ocean and Land Colour Imager (OLCI) instruments, OLCI-A and OLCI-B, which are aboard the Sentinel3A and Sentinel 3B satellites, respectively.
These data were compared to IFS predictions of spectral radiances, which were converted into reflectances. The images show some of the results of the comparison. The experiments performed in 2022 reveal the potential benefits of visible data assimilation. It could provide information ranging from the large scale, such as highlighting where regions of frontal clouds are located, to the smaller scale, such as revealing the structure of hurricanes or convective cells.
Observed and predicted reflectances of Hurricane Larry on three separate days
The top row shows reflectances from the OLCI observations, while the bottom row shows short-term IFS predictions. Each panel covers 15° in latitude and longitude.
ECMWF coordinates a European Horizon 2020 research project, CONFESS, that aims to improve climate reanalyses and seasonal forecasts by updating land and aerosol properties. CONFESS will evolve the capabilities of the EU-funded Copernicus Climate Change Service (C3S), run by ECMWF, to monitor and predict extreme events and represent climate trends. It is set to end in 2023.
In 2022, this project made progress in preparing a new record of land cover and vegetation observations. This is important to properly constrain the land surface models included in current reanalysis and seasonal-to-decadal prediction systems.
In particular, ECMWF produced a harmonised temporal record spanning the period 1993–2019 of Land Cover, Land Use and Leaf Area Index by merging records from C3S and the Copernicus Global Land Services.
CONFESS also produced a homogenous and consistent multi-decadal record of tropospheric aerosols. To do this, it exploited the atmospheric composition capabilities that the EU-funded Copernicus Atmosphere Monitoring Service (CAMS) has introduced into ECMWF’s IFS. This was done in preparation for ECMWF’s next reanalysis, ERA6, and its next seasonal forecasting system, SEAS6. Having an up-to-date aerosol climatology that is consistent with the latest CAMS aerosols will also help to benchmark the impact of interactive aerosols on numerical weather prediction.
Change in tropospheric aerosol between 1975 and 2015
The figure shows the change in vertically integrated black carbon (top) and sulphate (bottom) between July 1975 (left) and July 2015 (right). The effects of increased forest fires at high latitudes, pollution controls in Europe and North America, and the growth of emissions in India and the Middle East are all visible. Sulphate aerosols over China have peaked and are now declining.
ECMWF’s weather predictions rely on global weather observations to help determine the initial conditions of forecasts. A key question for the evolution of the global observing system is how much benefit we expect from new observing capabilities. One of the tools to investigate this is the Ensemble of Data Assimilations (EDA). The EDA estimates in a statistical sense the expected reduction of uncertainty in the forecast from adding observations to our forecast system.
The EDA has been used to assess the value of more passive microwave (MW) sounding observations. Currently, MW sounders are flown on a few large satellites. Results from the EDA experimentation show a clear continued benefit from adding further MW sounders.
An illustration of this is provided by the figure. It shows the reduction in short-range ensemble forecast spread of two constellations of eight hypothetical MW satellites (Polar) compared to the four existing MW sounders (Metop/JPSS) and a constellation without any MW sounders. The figure shows a very clear benefit of having temperature and humidity sounding channels available for the additional MW sounders.
Impact of additional MW satellites
Impact on short-range wind forecasts from the ‘Polar’ constellation (red), with humidity sounding and window channels only (dashed) and with temperature channels added (solid line), and the ‘Metop/JPSS’ baseline (black), relative to a ‘no MW sounder’ EDA experiment. Data are over (a) the northern hemisphere extratropics and (b) the tropics, for the period 8–28 June 2018.
An accurate knowledge of the ocean has a profound impact on our ability to forecast the weather over a variety of timescales. In particular, the sea-surface temperature (SST) needs to be determined extremely accurately to make successful forecasts. In 2022, we made progress towards a new way of determining SST.
Currently SST fields are produced by blending satellite information with in-situ ocean observations from ships and buoys. This depends on assumptions about how significantly the composition of the atmosphere has affected satellite radiation measurements, and on crude persistence assumptions when there are no observations at all.
These challenges prompted us to investigate the possibility of estimating SST inside our coupled atmosphere–ocean data assimilation system. That system combines all the latest observations with a short-range forecast constrained by previous observations to obtain the best possible estimate of the current state of the Earth system. It uses atmospheric 4D-Var data assimilation and ocean NEMOVAR data assimilation. A new system that has been developed to estimate SST is called RADSST.
Ocean temperature schematic
The schematic shows how ocean temperature can vary from the subsurface to the surface skin that is measured by satellites. An assumption in RADSST is that any mismatches between the parametrization of temperature in the surface skin (SKT) and the values sensed by radiance observations is attributed to an error of the same magnitude in the bulk water temperature below (SST).
Up until the end of October 2022, the two Cray XC40 supercomputing clusters in Reading, UK, continued to provide a good and stable service. On 18 October, our new high-performance computing facility (HPCF) in Bologna, Italy, comprising four Atos BullSequana XH2000 complexes, started to be used for operational production while acceptance was still ongoing. The move marked an important milestone in the BOND (Bologna Our New Data Centre) programme.
IFS Cycle 47r3 was successfully migrated to the new system after extensive testing demonstrated that the facility could provide a platform for current time-critical forecast production. Meanwhile, Atos continued to improve the HPCF’s performance and stability. With over 1,000,000 cores in the new facility, 25% of the supercomputing capacity is dedicated to Member States, of which up to 10% is reserved for Special Projects. The new HPCF will significantly increase the resources for these activities.
In addition to the standard ‘compute nodes’ for parallel jobs, each of the four complexes of the new HPCF has ‘GPIL (general purpose and interactive login) nodes’ for general purpose and interactive workloads. Also, one of the complexes includes a number of NVIDIA GPUs to support application development on accelerators.
ECMWF’s new HPCF
The Atos high-performance computing facility in ECMWF’s new data centre in Bologna, Italy.
In March 2022, the Copernicus Climate and Atmosphere Data Stores (CDS/ADS) run by ECMWF were successfully migrated from Reading to new cloud infrastructure installed in the Bologna data centre. This required careful planning and close collaboration, both with external contractors and between teams across ECMWF.
On the day of migration, the impact on users was kept to a minimum with a downtime of only five hours. After migration, the CDS/ADS performed well. The existing equipment in Reading was decommissioned and reconditioned. It was installed in Bologna at the end of June to further enhance the capacity of the data stores’ infrastructure.
Genetic algorithms (GAs) were deployed to tune the CDS/ADS in Bologna. GAs simulate evolution to find the best solution to a problem. They were used to find an optimal placement of virtual machines onto physical hosts in order to optimise the service offered by the CDS/ADS. This is an example of the increasing use of machine learning at the Centre, in this case to optimise the performance of a service.
Data delivered by the Climate Data Store
2022 saw a rise in the volume of data downloaded and the number of requests.
In mid-2022, Fugaku was the fastest supercomputer in the world with a sustained performance of 440 PFLOP/s. Fugaku relies on 160,000 Fujitsu A64FX CPUs, which use an ARM instruction set to achieve its extremely fast calculation speed. Fugaku CPUs also have an extremely wide vector register, of 512-bit, while they also natively support FP16 ‘half-precision’ floating-point arithmetic.
We were very interested to evaluate the scalability performance of the IFS on this type of architecture. The IFS was ported to Fugaku during the first half of 2022. The same IFS forecast was repeated by increasing the resolution and the number of nodes, so that the same amount of work was assigned to each processor.
The result is shown in the figure. The desired outcome is a line which is as horizontal as possible. The discrepancy that appears in the last datapoint shows a reduction in efficiency. This is probably due to a lack of sufficient memory in the current version of the Fujitsu chip on Fugaku. As this could be addressed by a chip with more memory, the outcome of this study was overall encouraging. It suggests that ARM CPUs could be considered for future ECMWF needs.
Scalability performance
The graph shows the scalability performance of the IFS on Fugaku. The x-axis shows the spectral truncation, with the equivalent horizontal grid resolution provided on top. The y-axis shows simulated days per day, measured from a simulation on Fugaku without output. All runs were performed up to ten days ahead.
In 2022, we started to make some of the IFS available on an open-source basis to facilitate collaboration on the code. A GitHub space was created to host open-source IFS components. The main aim of partially removing restrictions on redistribution was to make working with ECMWF more attractive to collaborators who wish to work with their partners.
Contributing to open-source codes could also be more attractive to academic partners. Another aim was greater efficiency as, for example, some journals require open access to the codes used. Finally, the move positioned ECMWF and our Member States at the centre of international efforts on emerging high-performance computing architectures. Making IFS code open source was deemed to encourage work by computational science experts in academia and vendors.
Discussions about the future approach were ongoing at the end of the year. The approach was to continue to release specific components of the IFS on demand and to periodically review whether this approach remained appropriate.
We developed a new Software Strategy in 2022, which sets out our plans for the development of software outside the weather prediction process until 2027. One of the guiding ideas is for software to be developed openly, with interaction and feedback from the community. Software components are also to be made smaller, more usable, and simpler to integrate with each other
The Software Strategy aims for a good balance between in-house development of software that is critical for ECMWF on the one hand, and well-maintained and supported community software on the other. It emphasises the need for improved scalability of data handling as the amount of forecast data continues to grow rapidly.
The turn towards the open development of new software is based on the belief that interaction and feedback from the community, including in particular our Member and Co-operating States, leads to increased quality. We have already had an open-source policy for all software not related to the IFS for many years. However, this did not enable continuous feedback and contributions from external users. To encourage greater collaboration, we aim to widen our use of GitHub and other open platforms.
Core data storage software
This is an example of the areas described in the Software Strategy. The MARS (Meteorological Archival and Retrieval System) ecosystem includes the FDB (Fields DataBase). In addition, ECFS (ECMWF’s File Storage system) provides an unstructured archive for data outside metadata-driven workflows.