Challenge 35 - Developing a Machine Learning-Based weather/climate Index for Compound Event Prediction in Renewable Energy Systems

[EsperanzaCuartero]

Challenge 35 - Developing a Machine Learning-Based weather/climate Index for Compound Event Prediction in Renewable Energy Systems

Stream 3 - Software Development for Earth Sciences Applications

Goal

Develop an innovative Machine Learning-based Compound Event Index (CEI) utilizing ERA5 reanalysis data to detect, analyze, and forecast (using IFS ensemble and the Climate DT/continous DT model) concurrent extreme weather events affecting renewable energy generation.

Mentors

Christoph Rudiger (ECMWF)
Irene Schicker, Annemarie Lexer (GeoSphere Austria)

Skills Required

• Strong Python programming skills
• Experience with ML frameworks (e.g., TensorFlow, PyTorch)
• Proficiency in handling large meteorological datasets
• Understanding of environmental science principles
• UI design experience beneficia
• Background in environmental sciences or related fields
• Interest in renewable energy systems
• Passion for applying ML to environmental challenges

---

Description

The transition to renewable energy faces a critical challenge: compound weather events. These are simultaneous occurrences of multiple adverse weather conditions that can severely impact renewable energy generation. When, for example, low wind speeds coincide with reduced solar radiation and drought conditions, the impact on power generation can be severe. Energy providers need sophisticated tools to anticipate these events for effective grid management and energy security planning to ensure that the energy demand is met.

Develop an innovative Machine Learning-based Compound Event Index (CEI) utilizing ERA5 reanalysis data to detect, analyze, and forecast (using IFS ensemble and the Climate DT/continous DT model) concurrent extreme weather events affecting renewable energy generation. The index should integrate multiple environmental variables such as evaporation, evapotranspiration, windspeed and direction, temperature, aerosol optical depth, and provide actionable insights for energy infrastructure planning.

1. Key Research Areas

• Compound Event Analysis
• Identify and quantify the co-occurrence patterns of extreme weather events
• Analyze temporal trends in compound event frequency and intensity
• Study spatial correlations and regional vulnerability patterns

2. Advanced Machine Learning Implementation

• Apply Self-Organizing Maps (SOMs) for weather pattern classification
• Develop anomaly detection algorithms for early warning systems
• Integrate ensemble forecasting techniques for uncertainty quantification

3. Environmental Process Integration

• Model soil-atmosphere coupling effects on energy generation (incl. analyzing land-atmosphere feedback mechanisms)
• Incorporate evapotranspiration and vegetation dynamics

4. Practical Application Development

• Create a generalised index calculation framework
• Implement regional adaptations using the Climate Data Tool (CDT)
• Develop user-friendly visualization tools for risk communication

---

Evaluation Criteria

• Feasibility
• Innovative approach
• Transferability
• Easy to maintain / Future-proof approach
• Matching requirements

[masterflorin]

Hello,

Sounds like a challenging topic, I have the impression that the outcome of this could also be a paper in a journal, besides the code for the index framework.

As far as I read, the dataset covers the period from 1940 to the present, with daily updates available five days behind real time, I'm assuming the dataset would be quite large even if you only a few variables. Do you have an idea what the time frame that will be used?

How are you supposed to use IFS ensemble for this project? Do you get access or receive some outputs via an API?

For the ML implementation, given the size of the dataset are there any compute resources available for training and inference?

Regarding the last point, could you clarify those three points? Namely, the framework as an outcome will be a repository of code, as for the visualizations, Jupyter notebooks with visualizations? Also, I've never used CDT before so I wouldn't want to make any assumptions regarding this one.

Florin

[ChrisR53175]

Hello Florin,

The entire data set is indeed large, but given the temporal density could well be regionalised while still ensuring a sufficient amount of data for the learning. ERA5 is available on GoogleEarth (e.g. https://developers.google.com/earth-engine/datasets/catalog/ECMWF_ERA5_LAND_HOURLY) and the idea is to use that resource for the data access.

The expected outcomes would be a trained model, including the visualisation tools, as notebooks, yes. The access to CDT can be facilitated and supported, where needed.

regards
Chris

[ischicker]

Hi Florin,

True, the data set is large but using the Google dataset (or arco-era5, https://github.com/google-research/arco-era5) is quite nice and you can start developing in a colab before downloading tons of data.
Also, one could start focusing on one region, limiting the size of the data and the resources for the ML implementation. Though, we might be able to provide something (maybe on the EWC).

The outcome is ideally a repo with code and visualisation code. Feel free to use jupyter for describing/documentation of the workflow.

cheers, irene

[m-schillinger]

Hi, I have a few questions:

• Are we expected to use also renewable energy data? If yes, what data / where does it come from / which variables?
• What spatial scale are we supposed to look at? Global, regional...? Or does this depend on the outcome or some of the first steps?
• Can you give a bit more background for part 3: what is the exact goal here? Are we expected to rely again on ERA5 data or incorporate also different data sources?
• More generally: Are proposals expected to follow the exact detailled steps above or how much freedom is there?
  Thanks a lot.

[ChrisR53175]

Hello Maybritt,

@ischicker will be best placed to respond to your first point.

On your further points:
2. The spatial extent can be regional, but it may depend on the variables you are incorporating. It will also depend on the type of renewables your focussing on. The important aspect is that there needs to be the ability to scale your approach to allow broader questions to be answered or other locations assessed, rather than just for the local ones the model was developed for.
3. Continuing on from the previous point, that also means that the data needs to be fairly consistent, accessible, and available over a longer period and preferably at global scales. Some local information may contain site-specific information, but won't be necessarily helpful if they are not available anywhere else. You would also need a reasonably long time-series to train your model - ERA5 is therefore one of the best sources. That doesn't mean that you only have to use ERA5, but you would need to consider the scalability when using other data sets.
4. No idea is perfect - there is room to propose variations from the steps above within the constraints of the overall idea of this project. Be creative and convince us :-)

regards
Chris

[ischicker]

Hi Maybritt,

as for the first point: you can use renewable energy data. As source, on a country level, you would have the freely available entso-e database dating back to, i think, 2014 with 15-minute resolution. Those are available for a number of renewable energy types and also distinguish onshore/offshore. You also get information on installed capacity per year. https://transparency.entsoe.eu/

regards, irene