© Crown copyright Met Office User requirements gathering – experience from the ESA Climate Change Initiative project on SST Simon Good

© Crown copyright Met Office The Climate Change Initiative

© Crown copyright Met Office The ESA Climate Change Initiative (CCI) “To realise the full potential of the long-term global Earth Observation archives that ESA, together with its member states, has established over the last thirty years, as a significant and timely contribution to the ECV databases required by United Nations Framework Convention on Climate Change (UNFCCC).”

© Crown copyright Met Office The ESA Climate Change Initiative (CCI) Each CCI project was required to produce a user requirements document.

© Crown copyright Met Office User requirements gathering

© Crown copyright Met Office Types of user requirements studies 1.Focus on the requirements of a particular group of users. Can make sure that the products really meet their specifications. It is likely that products will still also be useful to others. 2.Try to capture everyone’s requirements. Comprehensive list of requirements that will be a useful resource for the future. Useful to be able to show that our products meet the requirements of lots of people. There may be conflicting requirements and it is not possible to satisfy everyone.

© Crown copyright Met Office User requirements gathering For the SST CCI project we did a broad study. We were asked to gather requirements for a wide variety of aspects of SST data. Including getting feedback on current SST datasets and finding out user’s future needs. And at different levels of requirement: Threshold: the limit at which the observation becomes ineffectual and is no use for the application. Breakthrough: the level at which a significant improvement in this application would be achieved. Objective: the maximum performance limit for the observation, beyond which no significant improvement in the application would be achieved.

© Crown copyright Met Office Methods of user requirements gathering Review of documents produced by other projects/organisations. E.g. GCOS user requirements. Useful resource but don’t get to communicate directly with the users. Discussion sessions. Get a really good understanding of particular user’s needs. It is only possible to do this with a limited number of people. Asking other projects for lessons learned. Useful way to avoid pitfalls! Replies may be confidential.

© Crown copyright Met Office Methods of user requirements gathering Questionnaire. Good way to reach many people. This was the primary source of information for the SST CCI project user requirements. Getting ‘trailblazers’ to try the data and send feedback. Really good way to find out about good and bad things about the data. Only possible once data products are available.

© Crown copyright Met Office Experience from the questionnaire

© Crown copyright Met Office Getting people to fill out the online questionnaire Invites were sent to over 800 addresses. Mined from papers that cited SST datasets, and A list of climate research areas was compiled and contacts for each were found. The questionnaire was long! (30 minutes+ to complete.) There were 108 completed responses. At least as many started and then gave up!  Keeping a questionnaire short and focussed should maximise the number of complete responses.

© Crown copyright Met Office Demographic of responses

© Crown copyright Met Office Demographic of responses

© Crown copyright Met Office Demographic of responses

© Crown copyright Met Office Example result: length of record required A mixed set of results per application area. Is this because people’s use of the data is very different even within application areas? Or is it because people are not used to being asked to quantify things in this way?

© Crown copyright Met Office Example result: length of record required Clear increase in requirements going from threshold to breakthrough.

© Crown copyright Met Office Example result: length of record required Highest quality satellite SST data does not extend >30 years! Sometimes it is not possible to meet a requirement – can’t make apples out of oranges. But it is useful to capture these requirements as it could help identify opportunities for future work.

© Crown copyright Met Office Example result: grid spacing Some questions that are asked can directly influence the product specification.

© Crown copyright Met Office Example result: artificial drift Some requirements can be used to determine if the products meet user’s needs. However, in some cases it may be difficult or impossible to know because of a lack of reference data!

© Crown copyright Met Office Conclusions

© Crown copyright Met Office Conclusions For the SST CCI project we used a variety of methods to gather user requirements. An online questionnaire became the main source of requirements. Important to keep the questionnaire short and focussed. Answers to questions may: Be confusing and require further investigation. Directly influence product design. Help to benchmark the performance of products. Not be possible to achieve or not be possible to demonstrate that they are achieved because of limitations of the data/reference data. Getting trailblazing users involved can be very helpful.

© Crown copyright Met Office Questions and answers