Search for a command to run...
With the increase in simulation resolution, climate and weather models are now potentially outputting petabytes of data. The largest projects can thus require complex workflows tightly integrating pre-processing, computing, post-processing, monitoring, potential downstream applications or archiving. We introduce here Sirocco, a new climate and weather workflow tool written in Python in collaboration between ETHZ, PSI and CSCS with a special care for the ICON model. Sirocco is written with separation of concerns in mind, where users should only care about expressing their desired workflow and bringing the scripts/sources for each task independently. That's why "Sirocco" first designates a user-friendly yaml based configuration format. Inspired by cylc and AiiDA, it describes the workflow graph by equally integrating data nodes (input and output) alongside task nodes. Workflows thus become truly composable, in the sense that no task is making any assumption on the behavior of others.Sirocco currently defines two types of tasks, called "plugins". The "shell" plugin is dedicated to tasks for which users provide their own main executable, including any auxiliary set of files. The only requirement is the ability to interface with Sirocco, either with executables accepting command line arguments and environment variables and/or by parsing a yaml file providing the necessary context for task execution. The "icon" plugin is a dedicated user friendly interface to the ICON model. On top of the integration to Sirocco workflows, it provides easy ways of handling matters like date changing, namelist modifications, restart files or predefined setups for target machine and architecture. By design, other plugins can be written to facilitate the integration of any other application/model.Once an internal representation is generated from the configuration file, two possible back-ends can orchestrate the workflow. The first one, called "stand-alone", is entirely implemented inside Sirocco and runs autonomously on the target machine, only relying on the HPC scheduler daemon to keep the workflow running. The second one interfaces with the low-level workflow library AiiDA and its satellite packages, running on a dedicated server with its own daemon and dumping workflow metadata in a queryable database. Both orchestrators implement the novel concept of a deep dynamical task front that propagates through the graph, enabling the ahead-of-time submission of an arbitrary number of task generations.At the end of the day, Sirocco not only provides the ability to run complex workflows and a nice interface to ICON but also, through its workflow manager nature, facilitates shareability and reproducibility in the community.