COSMOS – A framework for containerised, distributed execution and analysis of hydraulic water distribution system models

Abstract

[EN] Many scientific problems related to water distribution systems like optimization problems or sensitivity analysis require the creation and execution of a large number of hydraulic models. To reduce computation times, different approaches have been used in the past, often by employing multiple CPU cores to solve the hydraulic equations of a single model or to simulate multiple models in parallel on a single computer. However, these approaches often cannot make use of distributed computing. Furthermore, using these approaches in applications with a (web-based) graphical user interface (GUI) often requires the development of tailored software solutions and application programming interfaces (API) to link GUI and model execution backend.To tackle these issues, we propose COSMOS (Containerised Model Simulator), a highly scalable Python-based framework which allows for the modification of hydraulic models using OOPNET, an API between Python and the hydraulic solver of the modelling software EPANET and can run model simulations. Simulation results can be then further analysed while all these tasks run encapsulated in containers in a cluster. It also allows to easily link the described functionality with other applications by providing a REST API.A standard-based OpenAPI allows for passing hydraulic models and running scientific workflows via HTTP and generating clients based on the provided OpenAPI schemas, which simplifies the creation of web-based user interfaces. Python was chosen because of its growing spread in the scientific community, the availability of data processing and optimization packages and its high code readability.Prefect, a data workflow orchestration framework, is employed to create workflows, starting with the transformation of hydraulic models into JSON representations for further use in web applications. The models can then be executed and simulated distributed over the available CPU cores (locally or in a cluster). Further tasks for doing analysis in the cluster can be easily added if necessary. Hydraulic models and simulation artifacts are stored on S3-compatible storage and can be easily retrieved.A main advantage of this approach is the use of containers, which allows for reproducible workflows. Compared to other high performance computing approaches and container-based systems, Prefect has the advantage of being able to keep dedicated worker nodes available for use. This comes in handy especially when dealing with relatively short computation times where the start of a container might take longer than the actual simulation. Additionally, simulation data post-processing can be easily added to workflows in Prefect. Furthermore, as the structure of COSMOS is highly scalable, it can be used for different levels of problem complexity and simulation runtimes.