Architecture

Framework Overview

_images/framework.png

ESGF is currently the main climate data resource (but more resources are possible). ESGF Solr-index is used to find ESGF data. The ESGF identity provider with OpenIDs and X509 certificate is used for authentication.

There are several WPS services. Malleefowl is the main one for the Phoenix client. Malleefowl is used to search, download (with caching) ESGF data and to retrieve certificates. Malleefowl has also a workflow engine (dispel4py) to chain WPS processes.

The results of the WPS processes are stored on the file system and are accessible via URL (with a token id).

Results can be shown on a Map using a Web Mapping Service (ncWMS, adagucserver).

The PyCSW Catalog Service is used to register WPS services and also to publish WPS outputs. Published results in the PyCSW can also used as input source for processes again.

WPS serivces can be accessed through web-applications like Phoenix or from scripts.

Note

See also the Birdhouse Presentation.

Birdhouse is the home of Web Processing Services used in climate science and components to support them (the birds):

Client Side Components

  • Phoenix: a web-based WPS client with ESGF data access
  • Birdy: a WPS command line client and native library

Server Side Components

WPS services for climate data analysis:

  • Flyingpigeon: services for the climate impact community
  • Black Swan: services for the extreme weather event assessments
  • Hummingbird: provides cdo and compliance-checker as a service
  • Emu: some example WPS processes for demo

Many climate analysis operations are implemented using OpenClimateGIS including the python package icclim.

Supporting Services and libraries:

  • Twitcher: an OWS Security Proxy
  • Malleefowl: access to climate data (ESGF, …) as a service
  • Eggshell: provides common functionallity for Birdhouse WPS services

You can find the source code of all birdhouse components on GitHub. Docker images with birdhouse components are on Docker Hub

Files and Folders

Warning

This section is outdated. We are moving to a new deployment without Buildout.

The birds have a similar folder structure. While library dependencies are stored within the conda deployment

Three folder locations have to be pointed out:

  • repository clones: The fetched code by git clone. It is recommended to store the repositories in ~/birdhouse
  • anaconda: By default, the installation process creates a folder ~/anaconda for general anaconda-specific software.
  • conda environments: All birds (repositories) are built with their own environment to avoid missmatch of dependencies. By default, the conda environments are in ~/.conda/envs/.

To change the default settings, create a Makefile.config with:

$ cp Makefile.config.example Makefile.config

and change the paths accordingly to your needs.

Furthermore, in environment.yml, the conda packages can be defined. It is recommended to pin the version. The bird-specific packages are defined here, while in requirements/conda_pinned, general versions are set.

There are log files situated at:: ~/birdhouse/var/log/pywps/