1. GSICS Product Submission & Promotion Documents 8.1 and 8.2
Manik Bali and Lawrence E. Flynn
2016 GSICS Executive Panel Meeting
Biot, France
June 3 , 2016
Document 8.1 & 8.2

2. Disclaimer
“The scientific results and conclusions, as well as any views or opinions expressed herein, are those of the author(s) and do not necessarily reflect the views of NOAA or the Department of Commerce."

3. Contents Introduction
8.1  Products recently promoted or submitted for promotion 
8.2 Lessons learnt + review of the promotion process 
Acceptance and Promotion of New Product Categories.
Conclusion and way forward
Certificate of Appreciation

4. Introduction
Since the last 7 years of our existence none of the products had been declared operational.
This is a grave situation because satellite life spans are limited, ( eg. EUMETSAT has recommended to switch over to IASI-B sooner rather than later).
Immediate steps should be taken to assign operational status to products after evaluating their basic goals and tenets have been met.
People have moved on to different projects while their products are operationally being produced and not used.
Non-operational products are difficult to market because users don’t have confidence in them. Use of data in publications can raise reviewers comments.
Action: GCC to take a lead in discussing the GPPA with GSICS members to reduce the amount of time needed to move through the phases.
-> In Delhi, 2015, GCC Proposed a GPPA that could be fine tuned to situation at hand and still maintained the QA4EO principals. Currently when a product is submitted several pathways are suggested to the producer from which the producer can pick the one that works best for him
Ref:

5. Product recently submitted or promoted
20 November 2018
Need EP advisory on three products
MSG 2/3 promotion to Operational
a) Need a formal EP approval – processes completed.
Acceptance of Prime Product.
a) Action + Need EP approval for acceptance + commitment from agency
3. Acceptance of DCC Product.
a) Need EP approval and commitment from agency to support product.
GIR e.1
Tim Hewison to consider revising terminology used in the current "Primary GSICS Corrections", during demonstration phase.
Kenneth Holmlund suggested that EUMETSAT also submit the following documents in support of the product becoming operational
GSICS_Impact_on_MTP_Products_Report
GSICS Meteosat-IASI Inter-calibration Validation Report for Meteosat-9 and -10
MPEF Alternative Calibration Coefficients from GSICS - Validation Report
Typically require 3 reviewers but with docs need 1 review.
These documents would help acceptance and promotion in GPPA and can be applied to all products submitted.
EP Advisory: Should we make these documents mandatory
Should we make the family of MSG Operational ? And NOAA GOES-IASI Products as well?

6. Petition to promote EUMETSAT MSG 2/3 – SEVIRI products to Operational status
On April Author Tim Hewison submitted a petition to assign MSG 2/3 SEVIRI –IASI products an Operational Status.
As per the laid out GPPA steps, the ATBD, Users Guide and Uncertainty document were sent to the GPAT members for review.
After the review, GPAT, GRWG and GDWG Chair recommended promotion to Op which was forwarded to EP.
EP subsequently approved the petition however recommended a review of the final steps. ( see latest draft of petition here )
Need formal EP-approval on making the product Operational
Visit Review Page

7. Additional Documents submitted by EUMETSAT
Kenneth Holmlund suggested that EUMETSAT also submit the following documents in support of the product becoming operational
GSICS_Impact_on_MTP_Products_Report
GSICS Meteosat-IASI Inter-calibration Validation Report for Meteosat-9 and -10
MPEF Alternative Calibration Coefficients from GSICS - Validation Report
Called
123 Documents
They serve the following purpose.
At the agency, provide justification towards creation of the product, this gives confidence to outside users about the product.
Help in GPPA as they directly provide at least one user feedback (1)
Ensures agency support towards creation and promotion of the product.
Proposal: Submission of such documents be made an integral part of GPPA. This would dramatically cut down the costs(effort) of product in GPPA.
Question: What is the GPPA maturity level when we need them: Demo Pre-Op or Op ?

8. EUMETSAT SEVIRI – IASI-A+IASI-B correction Product
On Sept 02, 2015 Author Tim Hewison submitted a petition to accept MSG 2/3 SEVIRI –IASI-A+IASI-B Prime Reference Product in Demo Phase.
GCC Initiated sent the product out for review and provided two options.
The product is reviewed by and recommended for acceptance by 3 reviewers
The producer submits the Impact and Validation reports ( 1 reviewer )
GCC received two favorable reviews.
Annual Meeting Action: Tim Hewison to consider revising terminology used in the current "Primary GSICS Corrections", during demonstration phase.
Question to EP : Should GSICS accept the product in Demo phase with/without the 123 documents.
Visit Review Page

9. EUMETSAT SEVIRI – MODIS DCC Product
On Sept 15, 2015 Author Sebastien Wagner submitted a petition to accept MSG2-SEVIRI –Aqua-MODIS Product in Demo Phase.
The product is based on published research work and has a wide acceptability in the VIS Group . For eg. A session on DCC approach was organized in Annual Meeting.( see here )
GCC suggested two options
Impact and Validation Reports + 1 review.
Three GPAT reviews.
Product should be reviewed within the VIS subgroup.
2 review received. Awaiting formal agreement from Dave Doelling ( Vis Chair) to review the product within the VIS-DCC members.
Question to EP : Should GSICS accept the product in Demo phase with/without the 123 documents.
Visit Review Page

10. Products submitted by KMA and ISRO
It is proposed that instrument of similar family are assigned GPPA maturity without going through the entire GPPA process
Principle has been applied in the past to GOES and MSG family while reaching Pre-Op Stage.
Challenge-> What constitutes a family ( GCC has an action on this).
Is the Meteosat 8/11 same family as Meteosat 9/10 ?

11. Adapting the GPPA to the various product categories
20 November 2018
QA4EO - Guidelines “
The Quality Assurance Framework for Earth Observation consist of ten distinct guidelines linked in the Guidelines Framework
Data Producer
(GPAF)
Data
Reviewer
(GPERF)
Data User
( GPERF)
GSICS Executive Panel
Acceptance is through meeting GSICS Goals.
Maturity is Acceptance+ satisfying user
Guidelines address
Quality Indicators
Traceability
Reference (measurement)Standard
Uncertainty
Ref: QA4EO The Guide

12. Adapting the GPPA to the various product categories
20 November 2018
Core Products: Bias products, new additions, products connected with ECV/FCDR
directly follow the GISCS Conventions.
GSICS Resources and Documents:
GPRCs, ICVS, Corrections not following GSICS convections but meeting GSICS goals.
Models and Data Sets:
GIRO, SBAF, Solar, Lunar, references, Intermediate Datasets used to create core products, Pre-Launch key data sets.
Tools:
SNO Matchup, Bias Monitoring, Display Tools, Readers, SBAF
Acceptance and Maturity of GPPA Can be applied
Acceptance and Maturity of the GPPA Can be applied Partly
Accepted in the subgroup.
One or all conditions to be fulfilled
Models/Data sets published and peer reviewed and internationally accepted.
Have users within the group.
LF: We will have to make it clear what products have passed which levels of scrutiny.
BI: Need to make clear to users and submitters which acceptance and maturity requirements apply to each submission.
LF: May be a table with one dimension components required for acceptance and the other dimension product classes and A or NA entries.
Accepted in the subgroup.
One or all conditions to be fulfilled
Models/Data sets published and peer reviewed and internationally accepted.
Have users within the group.
11/20/2018

13. Lessons learnt- Review of the promotion process
First time a petition was submitted to GCC for making operational.
Need a template…Feedback on petition…
Do we need a web-meeting to discuss the promotion to Operational status ?
Yes-> Brings everyone on board. GPAT members can share work and experience.
No-> GPPA is typically /form based. Puts pressure on GPAT members until the Pre-Op phase the product has obtained enough user feedback.
Do we need a review board ?
Currently we have a GPAT and/or external reviewers and Chairs and GCC Director who provide feedback to EP.
A GPPA for New Categories has been proposed. This is based on QA4EO

14. Way forward and Conclusion
GPPA has been made more efficient-> The EUMETSAT product has attained Operational status within a review of 6 months.
123 documents are an efficient way of minimizing costs of GPPA review .
Prime Product review successfully completed
DCC product received two positive reviews and followup reviews to take place within the VIS Group.
KMA and ISRO products GPPA underway.

15. Appreciation Certificates

16. Certificate of Appreciation
With the support of WMO and CGMS, we present this certificate to
Anke Kniffka
for her outstanding contribution to the review of GSICS MSG2/3-SEVIRI product. GSICS  
recognizes the extensive effort involved in this review that was completed at no cost to GSICS
Awarded: Jun
Certificate of Appreciation
Dr. Lawrence E. Flynn
Director GSICS Co-ordination Center
Dr. Jerome Lafeuille
WMO Representative to GSICS EP
Dr. Peng Zhang
Chair GSICS Executive Panel

17. Certificate of Appreciation
With the support of WMO and CGMS, we present this certificate to
Fangfang Yu
for her outstanding contribution to the review of GSICS MSG2/3-SEVIRI product. GSICS  
recognizes the extensive effort involved in this review that was completed at no cost to GSICS
Awarded: Jun
Certificate of Appreciation
Dr. Lawrence E. Flynn
Director GSICS Co-ordination Center
Dr. Jerome Lafeuille
WMO Representative to GSICS EP
Dr. Peng Zhang
Chair GSICS Executive Panel

18. Certificate of Appreciation
With the support of WMO and CGMS, we present this certificate to
Manik Bali
for his outstanding contribution to the review of GSICS MSG2/3-SEVIRI product. GSICS  
recognizes the extensive effort involved in this review that was completed at no cost to GSICS
Awarded: Jun
Certificate of Appreciation
Dr. Lawrence E. Flynn
Director GSICS Co-ordination Center
Dr. Jerome Lafeuille
WMO Representative to GSICS EP
Dr. Peng Zhang
Chair GSICS Executive Panel

19. Certificate of Appreciation
With the support of WMO and CGMS, we present this certificate to
Marianne König
for her outstanding contribution to the review of GSICS MSG2/3-SEVIRI product. GSICS  
recognizes the extensive effort involved in this review that was completed at no cost to GSICS
Awarded: Jun
Certificate of Appreciation
Dr. Lawrence E. Flynn
Director GSICS Co-ordination Center
Dr. Jerome Lafeuille
WMO Representative to GSICS EP
Dr. Peng Zhang
Chair GSICS Executive Panel

20. Certificate of Appreciation
With the support of WMO and CGMS, we present this certificate to
Masaya Takahashi
for his outstanding contribution to the review of GSICS MSG2/3-SEVIRI product. GSICS  
recognizes the extensive effort involved in this review that was completed at no cost to GSICS
Awarded: Jun
Certificate of Appreciation
Dr. Lawrence E. Flynn
Director GSICS Co-ordination Center
Dr. Jerome Lafeuille
WMO Representative to GSICS EP
Dr. Peng Zhang
Chair GSICS Executive Panel

21. Certificate of Appreciation
With the support of WMO and CGMS, we present this certificate to
Pete Francis
for his outstanding contribution to the review of GSICS MSG2/3-SEVIRI product. GSICS  
recognizes the extensive effort involved in this review that was completed at no cost to GSICS
Awarded Jun
Certificate of Appreciation
Dr. Lawrence E. Flynn
Director GSICS Co-ordination Center
Dr. Jerome Lafeuille
WMO Representative to GSICS EP
Dr. Peng Zhang
Chair GSICS Executive Panel

22. Certificate of Appreciation
With the support of WMO and CGMS, we present this certificate to
Peter Miu
for his outstanding contribution to the review of GSICS MSG2/3-SEVIRI product. GSICS  
recognizes the extensive effort involved in this review that was completed at no cost to GSICS
Awarded: Jun
Certificate of Appreciation
Dr. Lawrence E. Flynn
Director GSICS Co-ordination Center
Dr. Jerome Lafeuille
WMO Representative to GSICS EP
Dr. Peng Zhang
Chair GSICS Executive Panel

23. Certificate of Appreciation
With the support of WMO and CGMS, we present this certificate to
Randhir Singh
for his outstanding contribution to the review of GSICS MSG2/3-SEVIRI product. GSICS  
recognizes the extensive effort involved in this review that was completed at no cost to GSICS
Awarded: Jun 3, 2016
Certificate of Appreciation
Dr. Lawrence E. Flynn
Director GSICS Co-ordination Center
Dr. Jerome Lafeuille
WMO Representative to GSICS EP
Dr. Peng Zhang
Chair GSICS Executive Panel

24. Thankyou

25. Backup slides

26. Dear Manik and Larry
Thanks.  I have now posted your presentation on User requirements  (2nd version).
I found and corrected a typo in some places (FIDUCIO > FIDUCEO)
As concerns the presentation on product promotion it addresses 2 agenda sub-items, for which we have a total of 20 minutes:
8.1  Products recently promoted or submitted for promotion  (e.g. 5 minutes) This is where you should ask a decision from EP to approve the promotion of candidate products which have completed the GPPA under the supervision of GCC
8.2 Review of the promotion process  (e.g. 15 minutes) This is where you can raise  issues to improve the GPPA process.
In your slides these 2 aspects are mixed and your are mentioning a number of products at various stages - which don't have completed the process already - therefore the EP won't see clearly which decision is proposed to promote which  candidate products having reached which stage.
For item 8.1, could we have 1 or 2 slides maximum indicating clearly (e.g. in a table) which candidate products are compliant with the GPPA and are thus proposed by the GCC for EP decision to promote them  if relevant, or indicate that there is no proposal otherwise.  If the decision is clearly presented this agenda sub-item can be addressed in 3-5 minutes.

27. For item 8.2, I see two important points for discussion:
- the action on GCC (EP b)  updated one year ago, requesting to adapt the GPPA to the various product categories  (because the GPPA was defined for NRT corrections, but GSICS is now embracing a wider range of deliverables). Is there a proposal from GCC ?
- the discussion we had in January about the conditions for extending the approval to a "family" of products, which is also addressed by slides from EUMETSAT.
Would it be possible to streamline your presentation ? Too many details - including historical details which don't require discussion anymore - would make it very difficult to have a fruitful discussion by the EP in 15 minutes.
I will post your presentation as it is but would be happy to replace it later on if you have an updated version.
Regards
Jerome

28. Due: Two weeks after the Product enters the Pre-operational Phase.
Actual Review Process.
As per the GPPA , the following steps are to be fulfilled to make the product operational.
Send notification and GPAT Product recommendations to the Executive Panel regarding the Product . Who: GCC Director
Due: Two weeks after the Product enters the Pre-operational Phase.
Condition Satisfied: The GCC Deputy Director, Fangfang Yu, provide a notice of the promotion via GUMS to all members on 23 January The promotion of the product was also reported at the EP 14th Session in July 2013.
Executive Panel review of the GPAT recommendations. Executive Panel feedback regarding the product sent to the GCC Director.
Who: GSICS Executive Panel
Due: Six weeks after being notified
The Executive Panel did not provide any critical feedback on this product promotion.
3. GCC Director notifies the Product provider about the Executive Panel feedback.
Who: GCC Director
Due: Two weeks after receiving feedback from the Executive Panel
No feedback to report.

29. Due: Three months after entering the Pre-operational phase
4. Complete documents associated with GPAF Sections III.2.C (Analysis software documentation), III.2.D (Product version control plan), III.3.B (operations and distribution plan), and III.3.C (data user's guide) and submit the documents to the GCC.
Who: Product provider
Due: Three months after entering the Pre-operational phase
Condition Satisfied: The initial GPAF forms version control plan and distribution plan data users guide were submitted by the product developer to the GCC 30 April The GCC made them available via the GSICS wiki. The latest version are now at
5 . Examine the submitted documents (product version control plan, operations and distribution plan, and data user's guide).Who: GPAT
Due: One month after GCC received the documents
Condition Satisfied: The GPAT and GCC participated in a review process for the documents and iteratively provide feedback to the developers.

30. 6. Remediate any documentation and overall product issues following the Executive Panel and GPAT feedback Who: Product provider, GCC Director
Due: One month after receiving feedback
Condition Satisfied: The product authors have implemented GPAT and GCC recommendations and final document versions were made available on the wiki on 16 October 2015.
Pending: The Executive Panel may decide if there is a need to provide additional feedback on any issues as part of Item #8.
7. GPAT reviews the remediation material and decides if the requirements are now satisfied. Sends final recommendation to the GCC Director.Who: GPAT and GCC Director
Due: Thee weeks following conclusion of the document remediation period.
Condition Satisfied: Final reviews by three GPAT members have been received and the GPAT has recommended promotion to Operational status. Other documents were reviewed by the GPAT and GCC earlier during the Pre-operational promotion stage of review.
8 . GCC Director notifies the Executive Panel that the product has satisfied all the requirements for entering the Operational Phase .
Who: GCC Director
Due: One week following the GPAT review.
Condition Satisfied: This memo serves as notification that the subject product has satisfied all requirements for entering the Operational Phase.
Pending: We await the Executive Panel direction.

31. Lessons learned
First time a petition was submitted to GCC for making operational.
Need a template…Feedback on petition…
Do we need a web-meeting to discuss the promotion to Operational status
Yes-> Brings everyone on board. GPAT members can share work and experience.
No-> GPPA is typically /form based. Puts pressure on GPAT members. Till the Pre-Op phase the product has obtained enough user feedback.
Do we need a review board ?

32. EUMETSAT – DCC Product
The SEVIRI- MODIS- DCC product submitted to GCC on 15 Sept 2015.
Fill out Sections I, II, III.1.A.1, and III.1.A.2 of the GSICS Product Acceptance Form (GPAF). Also, fill out Section III.1.B (ATBD) of the GPAF, but during the Submission Phase the ATBD needs only to be a preliminary version. The ATBD could be a journal article, technical memorandum or other documentation of the method used to make the product.
Who: Product provider
Done
Submit the GPAF and preliminary ATBD for review to the GSICS Product Acceptance Team (GPAT) via the GSICS Coordination Center (GCC) Deputy.
Determine if the GPAF is filled out correctly and decide if the product theoretical basis is adequate and the product scope is within the GSICS domain.
Who: GPAT
Due: 6 weeks after the GPAF submission
GPAT feedback given to the Product provider.
Who: GCC Director
Due: 7 weeks from GPAF submission
Upload a sample file to a GSICS data server if GPAT feedback is affirmative.
Who: Product provider, GDWG Chairman
Due: One week after GPAT feedback regarding GPAF form submission
File upload location
7. Remediate any GPAF issues according to the GPAT feedback. Who: Product provider, GCC Director
Due: Within 20 days after sending GPAT feedback to the Product provider

33. KMA – COMS –IASI product
Fill out Sections I, II, III.1.A.1, and III.1.A.2 of the GSICS Product Acceptance Form (GPAF). Also, fill out Section III.1.B (ATBD) of the GPAF, but during the Submission Phase the ATBD needs only to be a preliminary version. The ATBD could be a journal article, technical memorandum or other documentation of the method used to make the product.
Who: Product provider
Done
Submit the GPAF and preliminary ATBD for review to the GSICS Product Acceptance Team (GPAT) via the GSICS Coordination Center (GCC) Deputy.
Determine if the GPAF is filled out correctly and decide if the product theoretical basis is adequate and the product scope is within the GSICS domain.
Who: GPAT
Due: 6 weeks after the GPAF submission
GPAT feedback given to the Product provider.
Who: GCC Director
Due: 7 weeks from GPAF submission
Done- > New data file coming soon.

34. KMA – COMS –IASI product
Upload a sample file to a GSICS data server if GPAT feedback is affirmative.
Who: Product provider, GDWG Chairman
Due: One week after GPAT feedback regarding GPAF form submission
Pending
Determine if the sample file follows the GSICS NetCDF and file naming conventions.
Who: GDWG Chairman
Due: 2 weeks after successful product sample file upload
Remediate any GPAF issues according to the GPAT feedback.
Who: Product provider, GCC Director
Due: Within 20 days after sending GPAT feedback to the Product provider
Once all issues are resolved, the product enters the Demonstration Phase.
Who: GCC Director in consultation with the GPAT
Due: Within 90 days of the GPAF submission

35. ISRO-INSAT- IASI
Response from Producer awaited after first GPAT review.
Producer indicated problems with getting reference IASI –A data sets.

36. Conclusion and way forward
The GSICS Procedure for Product Acceptance (GPPA) has been applied on products from EUMETSAT, KMA and ISRO.
Product MSG 2/3 –SEVIRI have become operational- > Family of MSGs too can come in the same ambit and be declared operational.
Review of Prime Product provided by JMA and NOAA.

38. THANK YOU

39. MSG 2/3 Reviewers Anke Kniffka Fangfang Yu Manik Bali Masaya Takahashi
Marianne König
Peter Miu
Pete Francis
Randhir Singh