Climate Simulation Platform

Wesley Banfield

Plan

  1. Introduction
  2. Overview of Graphical User interface
  3. Project perspectives
  4. Technical discussion around implementation

Introduction

The Goal

  • The simulator needs multiple files as inputs
    • Topography
    • Routing (flow directions, ...)
    • Plant functional Types
    • Heat Flux
    • Weights between ocean and atmosphere
    • ...
  • These files are generated sequentially from one initial file (Paleo Topography)

State of the Art

Current workflow?

  • Collection of Scripts and Routines
    • Have to recompile and run manually in the correct order
    • Multiple versions exist causing inconsistencies between users
  • Cumbersome workflow for editing files

The Idea

  • Single source of truth → Deployed on shared server
  • Interactive file editing → Uses Javascript for interactivity
  • Automated execution → Runs scripts in background in correct order

→ Climate Simulation Platform

A web based tool to setup boundary conditions

Graphical User Interface

Login

  • Provides a personalized workspace
  • Two login methods:
    • Anyone can register → Good for local deployments
    • Only Registered people can add people → Better for cloud deployments

Login

Workflow

Additional Features

Perspectives

  • Move current application "up one step" and use interface for different tasks
  • Create different Boundary conditions tools (other than CMA5A)
  • Add post processing

Non Technical Questions?

Not interested in the technical stuff?? You are free to go

Under the hood

Microservice architecture (containers)

flowchart TD A[User Interface] <--> F[Reverse Proxy] <--> B[Message Broker / Message Dispatcher] B <--> C[Worker1 eg. Python] & D[Worker2 eg. Mosaic] & E[WorkerN eg. ...]

Docker

  • Conceptually scriptable Virtual Machine → Run workflow automatically
  • Each containers has its own environment → Run Python, Fortan, bash, Julia ....
  • Run multiple instances of same container → Scalability
  • Relatively simple to deploy → Facilitated CICD

Interactive data manipulation (holoviz)

  • Use pangeo stack eg. xarray + holoviz
  • All graphs use Holoviews with Bokeh for interactivity (written in python and translated into javascript)
  • Single page webapps (Internal Oceans, Diffusive Passages, ... ) use panel provided by holoviz

Algorithms

  • Routing (heavily inspired by Jean Baptiste Ladants work) see Multipage_webapp/python_tools/climpy/bc/ipsl/routing.py
    • Cyclicity ensured by padding arrays
    • We use skimage.morphology.reconstruction to fill depressions (internal oceans → plains) Always applied even if no internal oceans visible
    • Front migration algorithm used to add gradient towards exit points of plains
    • Trip calculated with matrix calculations (difference in 4 Directions) and take biggest difference though the 8 2D arrays

Algorithms

  • Heatflow (Rewritten version in python of Fred Fluteau's code) see Multipage_webapp/python_tools/climpy/bc/ipsl/heatflow.py
  • Diffusive passages and masks
    • We use convolutions scipy.signal.convolve2d
    • Inspired by image processing
    • Pass a template (generally small ~ 3x3) over the data to pick out desired patterns
      ? Water ? 0 1 0
      Land Water Land → -1 1 -1
      ? Water ? 0 1 0

Messaging

Why?

  • Communication between the different services
  • Automation of tasks (linked tasks)

Messaging

How? → RabbitMQ message broker

  1. When a button is clicked a message is sent to RabbitMQ eg. preprocessing.routing
  2. The message is picked up by the "Dispatcher" and "tagged" with the correct worker. eg. preprocessing.routing.python
  3. The correct worker picks up the message from the queue with the parameters and executes the job
  4. When the job is done the worker sends the message with a .done appended, eg. preprocessing.routing.done
  5. This is picked up by the dispatcher and the dispatcher sends the dependant tasks
graph TD UI(UI) --> | Task 1 | M(Message Dispatcher) M --> w1(Worker 1) M --> |Correct worker for Task 1 | w2(Worker 2) M --> wn(Worker N) w2 --> |Task1.done | M3(Same Message Dispatcher) M3 --> |Calculates dependant tasks if any | M2(Same Message Dispatcher) M2 --> |Correct worker for Task2 |w21(Worker 1) --> M3 M2 --> w22(Worker 2) M2 --> w2n(Worker N)

Storage

  • Persistent Storage
  • SQLite file → Simplest
  • Data stored as files rather than binary blobs → Cleaning up needed but faster Database
flowchart TD B[UI] <--> A[SQLite Database] C[Worker1] <--> A D[Worker2] <--> A E[WorkerN] <--> A F[Panel App] <--> A
classDiagram class User{ id Username Password } class Data_files{ id owner_id filename longitude latitude info } class Revisions{ id datafile_id created filepath revision file_type info } class Steps{ id data_file_id step parameters up_to_date }

Reverse Proxy

There is an nginx reverse proxy to redirect traffic between the flask app (General UI) and the panel apps (interactive editing)

Continuos Integration Continuos Deployment (CICD)

  • Tasks run on certain criterion (see .github folder in source)
  • On pull requests:
    • Run unit tests on Python Code
    • Build Docker containers and Push to DockerHub and OSUPytheas Registry
  • When a new image is pushed to OSUPytheas, Watchtower sees this, stops (gracefully) the current container(s) with the same image, then redeploys the new image with the same command line options

Portainer

Deployment of the Application Stack to a publicly accessible IP meaning the community can use the tool

Graciously put to our disposition by SIP OSUPytheas Person to contact Julien Lecubin

Documentation

Documentation is written on the gh-pages branch and in the docs folder in markdown, this is automatically translated using Jeykll and deployed to https://cerege-cl.github.io/netcdf_editor_app/

Perspectives

  • Create a login using LDAP or CNRS logins
  • Integrate MOSAIX
  • Interact with HPCs to send data to and from the effortlessly

Thank You

It's been a blast 🚀

Questions?