An update on BFG, The Bespoke Framework Generator Graham Riley (& Rupert Ford, STFC) Coupling Workshop Boulder, Colorado - February 20 th -22 nd 2013

Overview Why BFG? What is BFG? Not a coupler! What is new? - New implementation in python, faster  Interface to other languages (‘R’ and GAMS in IAMs) - Component- and program-compliance - Portal and BFG utilities  the power of metadata! - Parallel model support (and Infrastructure API) - Model ‘export’ – step to interoperability...  to couple to existing models using other technologies, such as ESFM or OASIS References & Future

Why BFG?  Community-based coupled modelling (still) emerging: From Centres of expertise… - in and across science domains (e.g. ESM, IAM, Flood modelling etc.) Distributed across institutions Implied: need for rapid exploration of use of models; sharing of models etc.  Requirement for flexibility to choose appropriate models

Why BFG?  This historic development of science has led to several coupling technologies (infrastructures/frameworks): Shared memory, MPI, TDT, MCT, OASIS, ESMF, CESM, FMS… (and other infrastructure, e.g. for parallel models etc.)  Requirement for flexibility to choose appropriate supporting technology Similar situation for the community which led to MPI standard?

Why BFG?  Choice of computing systems: Many options… - (Parallel) laptop/desktop, cluster, cloud, supercomputers… Rapid developments in hardware technology  Requirement for flexibility to use available computing systems efficiently

What is BFG? Aims…  Isolation of science code (i.e. models) Protect science code from changes in supporting software technologies and hardware (as far as possible)  Flexibility in composition and deployment Allow models to be put together rapidly to create new coupled models (as science permits!) Allow the models to be deployed into program units to make best use of computing resources Allow the coupled model to use the most appropriate existing coupling technology (or technologies)  Provide this flexibility through metadata descriptions and code generation (BFG2)  See other benefits of metadata-based approach…

What is BFG2?  A metadata description of a coupled model Capturing both its static structure and run- time, dynamic behaviour. The individual models described have to conform to certain, minimal, rules (see next).  A set of tools that use the metadata BFG2 - generates ‘wrapper’ code that can be compiled with model code to build an implementation. - Using a choice of existing (coupling) technologies etc. Visualise the coupled model (static composition and dynamic – control schedule – behaviour) Other utilities – see the BFG portal later…

What is a BFG2-compliant model 1.Component compliance (fine-grain models) Essentially a subroutine (in Fortran) Science code communication - by argument or ‘in-place’ put() and get() Control ‘outside’ the model 2.Program compliance A model already in an existing program Communication by put() and get() Other BFG calls (e.g. end_of_step()) Possibly embedded control  (Support composition of fine-grain models and aggregation of program-based models

A component-compliant model module atmos use bfg, only : put,get implicit none private public :: init,iteration,finalise contains subroutine init(arg1,arg2,...) end subroutine init subroutine iteration(arg1,arg2,...) call get(data3,tag3) call put(data4,tag4) end subroutine iteration subroutine finalise(arg1,arg2,...) end subroutine finalise end module atmos Couping by argument Tags generated by BFG Couping by in-place put()/get() The ‘tag’ connects to the metadata! Key to BFG ‘Standard’ init, run, finalise

10 BFG2 model ‘wrapping’ Container, Control (“Driver” layer), ArgPass Data, ArgPass Wrapper Model Code(s) Concurrency InPlace Calls Target Coupling Infrastructure - Existing code/library code - BFG-generated code  Component-compliance

Flexible Deployment atm sea ice chem ocean bio-geo model program unit/deployment unit T T T T transformer

A program-compliant model program atmos use bfg, only : put,get, bfg_init, bfg_eos, bfg_finalise implicit none call bfg_init() call get(data1,tag1) call put(data2,tag2) do i=1,nts call get(data3,tag3) call work(...) call put(data4,tag4) call bfg_eos() end do call get(data5,tag5) call put(data6,tag6) call bfg_finalise() end program atmos Stand alone program Only put()/get() in the ‘run’ section Calls to tell BFG about control structure Generated ‘target’ technology init.

Overview of BFG2 metadata  Model(s) Name, language Type - Science/transformer Coupling data - As args: in, out, inout - As put/get timestep  Composition Coupling description - What connects to what Priming data - Which connections are first started - Controls sequential or concurrent model execution  Deployment Mapping of models to ‘threads of control’ and main programs - to sequence units (“this model runs before that” in a ‘thread’) - to deployment units (-> the sequence units in a main program) Schedule - Iteration control of model execution  Nested loop structures allowed Target coupling technology All together, the metadata captures the run-time behaviour of the coupled model Think of SPMD model

Output Python engine Input xml script coupled xml Deployment xml Composition xml model Description xml xslt Templates & xslt Code and scripts What is BFG2? - summary  Metadata + generative approach

The Power of Metadata - BFG Portal  For examples of the use of metadata Prototype, in development bfg.cs.man.ac.uk/ Demos available on request!  Several utilities exist Validation Upload metadata and run BFG Access to some (small) examples Visualisation of composition and control/schedule Model stub generation (for test purposes) Makefile generation …

Simple ESM… Composition and Schedule views

Future?  Trend towards a common, shared infrastructure for ESM (and beyond)? Similar situation for the community which led to MPI?  And/Or – go for Framework interoperability BFG export is a first exploration (we have a design for BFG import too)  Use of (and extension of) CIM metadata Out of METAFOR project More uses of metadata - Including support for performance analysis, for example.  BFG as a Domain Specific Language…

Web References  BFG web page – including papers  BFG portal:  A BFG Wiki (from the GENIE project)  IS-ENES,EU project deliverables D8.3, “Towards Flexible Construction of ESMs using BFG”, March 2012 D8.5 – “ditto”, final report, March 2013

Thanks to our sponsors…

Towards parallel models (D8.5)  Support for parallel models on quasi-uniform grids  Wider than coupling Looking at other infrastructure support too - E.g. halo exchange (from MCT and ESMF)  Prototype “manUgen” solution Mainly targeting MCT - but some ESMF and OASIS3-MCT (for coupling) tests  Developed a prototype Infrastructure API Metadata + configuration of run-time system approach - plus some code generation

Infrastructure API (D8.5)  A minimal API for use by model developers  Metadata-driven  Target coupling and halo exchange  Support implementation of API in any of a number of existing technologies  Select ‘best of breed’

Parallel Model Support (D8.5)

Code Generation vs Configuration (D8.5) Metadata describing coupling and partition etc. Per model ModelInfo structure captures specific requirements – for halo exchange and coupling etc. Models use ‘tag’ in calls to access ModelInfo

Composite models – BFG2 export (D8.3) BFG atmos components dyn rad conv Ocean (OASIS) Ocean (ESMF) Atmos (OASIS) Atmos (ESMF) Export to ESMFExport to OASIS Coupling via OASIS Coupling via ESMF Existing Ocean models

ERMITAGE use case GENIE ‘fast’ ESM model (or emulator) LPJmL – veg./hydrology model MAgPIE – land use model TIAM/REMIND – enegy/economic models

An IAM example

‘Import’ of models  Aim: to enable models written to use different existing coupling technologies to talk to each other Using a selected communication mechanism…

BFG import Ocean (OASIS) Coupling via OASIS BFG Adaptor Coupling via BFG- generated ‘target’ ? Atmos (ESMF) Coupling via BFG- generated ESMF Atmos (ESMF) BFG Adaptor Coupling via BFG- generated TARGET Target could be: (almost) anything BFG will support… … MPI, GridMPI, OASIS3/4, ESMF, web services… BFG-generated ‘adaptor’ code E.g. web services Import both models

BFG import… Ocean (OASIS) Coupling via OASIS BFG Adaptor Coupling via BFG- generated ‘target’ Coupling via BFG- generated ‘target’ Atmos (BFG-gen)

Summary  Working with real ESM codes takes a lot of effort Experience with GENIE and JULES  Using ‘toy’ models to demonstrate capability and potential  Motivation from current ESMs is small Few models (but growing number) Large cost of change  Seeking use cases for export and import Examples of community collaboration (c.f. NEMO) Case for import made in ERMITAGE project  An emerging infrastructure requiring long term support for the community