Architecture

Framework Overview

_images/framework.png

ESGF is currently the main climate data resource (but more resources are possible). ESGF Solr-index is used to find ESGF data. The ESGF identity provider with OpenIDs and X509 certificate is used for authentication.

There are several WPS services. Malleefowl is the main one for the Phoenix client. Malleefowl is used to search, download (with caching) ESGF data and to retrieve certificates. Malleefowl has also a workflow engine (dispel4py) to chain WPS processes.

The results of the WPS processes are stored on the file system and are accessible via URL (with a token id).

Results can be shown on a Map using a Web Mapping Service (ncWMS, adagucserver).

The PyCSW Catalog Service is used to register WPS services and also to publish WPS outputs. Published results in the PyCSW can also used as input source for processes again.

WPS serivces can be accessed through web-applications like Phoenix or from scripts.

Note

See also the Birdhouse Presentation.

Birdhouse is the home of Web Processing Services used in climate science and components to support them (the birds):

Client Side Components

  • Phoenix: a web-based WPS client with ESGF data access

  • Birdy: a WPS command line client and native library

Server Side Components

WPS services for climate data analysis:

  • Emu: some example WPS processes for demo

  • Flyingpigeon: Testbed for new process development

  • Black Swan: services for the extreme weather event assessments

  • Hummingbird: provides cdo and compliance-checker as a service

  • Finch: services for climate indices calculation

  • Kingfisher: Services for Earth-Observation data analysis

Many climate analysis operations are implemented using OpenClimateGIS including the python package icclim.

Supporting Services and libraries:

  • Twitcher: an OWS Security Proxy

  • Malleefowl: access to climate data (ESGF, …) as a service

  • Eggshell: provides common functionallity for Birdhouse WPS services

You can find the source code of all birdhouse components on GitHub. Docker images with birdhouse components are on Docker Hub

Files and Folders

This is an overview of folder structure and important files for administration of a server-side birdhouse ecosystem.

It is recommended to clone the separated WPS services (birds) into one top level folder like:

$ ~/birdhouse/emu
$ ~/birdhouse/pyramid-pheonix
$ ~/birdhouse/finch
$ ~/birdhouse/malleefowl
...

The dependencies of each bird is deployed as conda environment and per default located at:

$ ~/.conda/envs/

The environment of a bird is defined in ./{birdname}/environment.yml.

Process descriptions are placed in ./{birdname}/{birdname}/processes/ while modules designed and used for the service are situated in ./{birdname}/{birdname}/. Here are also static data like shapefiles, templates or additional data used by the processes.

$ ./{birdname}/{birdname}/data/shapefiles
$ ./{birdname}/{birdname}/templates

Each birdhouse compartment has a documentation build with Sphinx and the corresponding files are situated in

$ ./{birdname}/docs

When running a service, files and folders for input data, result storage, file cache of simply logfiles are defined in the ./{birdname}/.config.cfg. Default configuration is defined in ./{birdname}/{birdname}/default.cfg as well as an example can be found in ~./{birdname}/etc. For more options of configuration see the pywps configuration instructions

For development and deployment testing the installations be checked running tests (make test). Test descriptions testdata are situated in:

$ ./{birdname}/tests
$ ./{birdname}/tests/testdata