OE Pre-award Six-month Evaluation Results March 2012 Executive Summary OE Pre-award Implementation Team 1.

Slides:



Advertisements
Similar presentations
Project Management Concepts
Advertisements

CHAPTER 7 Business Management.
HR Manager – HR Business Partners Role Description
Comprehensive Organizational Health AssessmentMay 2012Butler Institute for Families Comprehensive Organizational Health Assessment Presented by: Robin.
How to Enhance Personal Productivity By Janet Hadley
Performance Management Guide for Supervisors. Objectives  Understand necessity of reviews;  To define a rating standard across the Foundation for an.
University of Florida Academic Activities Reporting (Formerly IWF)
Measurement. T EAM STEPPS 05.2 Mod Page 2 Measurement Objectives  Describe the importance of measurement  Describe the Kirkpatrick model of training.
Leading Teams.
POST APPROVAL MONITORING AND EDUCATION PROGRAM Jane Lehmbeck 1, Sarah Blackman 1, Karen Davenport 2, Karen Parks 1 Office of the Vice President for Research.
MODULE 2. CUSTOMER SERVICE By Dale Pfeiffer. Session Learning Objectives Customer Service Understand the Basics of Customer Service Understand the 7-Steps.
1 Introduction to Workforce Planning and Development in State of Alaska Executive Branch Departments.
1 Academic Senate May 16, 2011 Pre-award Implementation The Principles Proposed Governance Performance/Metrics Programmatic Groupings for Pre-award Teams.
PPA 502 – Program Evaluation Lecture 10 – Maximizing the Use of Evaluation Results.
Empowering Staff Through Institute Planning (ESTIP) Executive Workshop Institute Name: XXXXXX Presenter: XXXXXX Date: XXXXXX.
By Saurabh Sardesai October 2014.
Quality evaluation and improvement for Internal Audit
Center for Health Care Quality Licensing & Certification Program Evaluation 1 August 2014 rev.
SAS Project Insight Planning Session 1 Good Morning!
Purpose of the Standards
2010 Performance Evaluation Process Information Session for Staff
Human Resources Office of Summary of Results 1 University of Minnesota Morris.
Stanford Employee Survey. Topics Background, Purpose of Employee Satisfaction Surveys –Survey Specifics –Survey Participants, Spring and Fall 2010 Fall.
Info-Tech Research Group1 Improving Business Satisfaction Moving from Measurement to Action.
Grants Business Process Re-Engineering (BPR) Overview
Federal Consulting Group August 2004 Department of Labor Civil Rights Center 2004 Satisfaction Study - Recipients.
Community Planning Training 1-1. Community Plan Implementation Training 1- Community Planning Training 1-3.
OE PRE-AWARD UPDATE SOM FACULTY COUNCIL MARCH 14, 2013 Susanne Hildebrand-Zanki - AVC, Research 1.
Final Client Briefing Call Center Wait Time Project August 27, 2012 – December 11, 2012 State of Florida – Department of Economic Opportunity Florida State.
PMP® Exam Preparation Course
Organization Mission Organizations That Use Evaluative Thinking Will Develop mission statements specific enough to provide a basis for goals and.
McLean & Company1 Improving Business Satisfaction Moving from Measurement to Action.
UBC Department of Finance Campus Community Customer Service Survey Forum Presentation March 1, 2004.
Using Metrics to Drive Research Administration Performance
Fundamentals of Evaluation for Public Health Programs ROBERT FOLEY, M.ED. NIHB TRIBAL PUBLIC HEALTH SUMMIT MARCH 31,
Certificate IV in Project Management Introduction to Project Management Course Number Qualification Code BSB41507.
BPR PHASE IV IMPLEMENTATION NOAA Business Process Re-Engineering Current Status: June 21, 2006 NESDIS Cooperative Institute Directors and Administrators.
OE PRE-AWARD UPDATE SOM FACULTY COUNCIL MEETING JUNE 21, 2012 Susanne Hildebrand-Zanki - AVC, Research 1.
Human Resources Office of FACULTY Summary of Results _College of Design_FACULTY.
2005 Performance Development System Survey Human Resources Staff Meeting March 20, 2006.
1. Housekeeping Items June 8 th and 9 th put on calendar for 2 nd round of Iowa Core ***Shenandoah participants*** Module 6 training on March 24 th will.
Georgia Institute of Technology CS 4320 Fall 2003.
Take Charge of Change MASBO Strategic Roadmap Update November 15th, 2013.
AN INTRODUCTION Managing Change in Healthcare IT Implementations Sherrilynne Fuller, Center for Public Health Informatics School of Public Health, University.
1 EMS Fundamentals An Introduction to the EMS Process Roadmap AASHTO EMS Workshop.
Performance Management A briefing for new managers.
Segment 6: Provider Communication California ICD-10 Site Visit Training segments to assist the State of California with the ICD-10 Implementation June.
Systems Accreditation Berkeley County School District School Facilitator Training October 7, 2014 Dr. Rodney Thompson Superintendent.
Project management Topic 7 Controls. What is a control? Decision making activities – Planning – Monitor progress – Compare achievement with plan – Detect.
European Social Fund Promoting improvement Shirley Jones.
Catholic Charities Performance and Quality Improvement (PQI)
2015 SAA Board Survey. Raw Board Survey ResultsStrongly Agree AgreeDisagreeStrongly Disagree Don't Know Total Points Responses minus DKs Average Score.
The Implementation of BPR Pertemuan 9 Matakuliah: M0734-Business Process Reenginering Tahun: 2010.
Evaluate Phase Pertemuan Matakuliah: A0774/Information Technology Capital Budgeting Tahun: 2009.
1 An Overview of Process and Procedures for Health IT Collaboration GSA Office of Citizen Services and Communications Intergovernmental Solutions Division.
Info-Tech Research Group1 Manage IT Budgets & Cost World Class Operations - Impact Workshop.
BSBPMG501A Manage Project Integrative Processes Manage Project Integrative Processes Project Integration Processes – Part 2 Diploma of Project Management.
Human Resources Office of 1 Summary of Results College of Design Dean’s Reports.
1 QUALITY MANAGEMENT SYSTEM PRESENTATION TO BOTSWANA DRUG ADVISORY BOARD MEMBERS 13 th – 17 th AUGUST 2007.
2015/16 Staff Performance Appraisals Webinar for ANR Supervisors Spring 2016.
Jayne Schaefer, BA Workforce Programs Manager Mather LifeWays Evanston, Illinois Toward Building a Sustainable Long-Term Care Workforce: LEAP.
HOW TO START AN INTERNSHIP FOR COLLEGE STUDENTS JIM KENNEY DSHS/ALTSA/HCS FEBRUARY 11, 2014.
Shared Services Initiative Summary of Findings and Next Steps.
Welcome. Contents: 1.Organization’s Policies & Procedure 2.Internal Controls 3.Manager’s Financial Role 4.Procurement Process 5.Monthly Financial Report.
TELL Survey 2015 Trigg County Public Schools Board Report December 10, 2015.
How Can Non-Profit Evaluators Best Support Field Staff in Data Collection? American Evaluation Association Annual Conference October 19 th, 2013 Kelly.
Tell Survey May 12, To encourage large response rates, the Kentucky Education Association, Kentucky Association of School Administrators, Kentucky.
UCL Annual Student Experience Review
Finance & Planning Committee of the San Francisco Health Commission
2018 Great Colleges Survey for Champlain College
Presentation transcript:

OE Pre-award Six-month Evaluation Results March 2012 Executive Summary OE Pre-award Implementation Team 1

Contents Introduction Summary of Phase 1A results Client satisfaction results Quality of work results Internal processes results Cost results Training results Implementation Challenges and Keys to Success Acknowledgments Appendix: Methods 2

Introduction This six-month evaluation measured and evaluated the implementation, operations and performance of the Phase 1A Team of Research Management Services (RMS). Presented here are significant and useful findings from the evaluation. The purposes of the six-month evaluation were to: Guide continued successful implementation of RMS Establish an accurate record of organizational development as it occurs Provide data for decision-making in on-going operations Benchmark progress toward achievement of goals 3

Introduction- cont'd The Phase 1A evaluation period is 8/15/11- 2/14/12 The original Phase 1A RMS Team included: 1 Team Manager 8 RSCs 2 Associate RSCs The 12 Phase 1A department clients: 4 Cell and Tissue Biology Clinical Pharmacy Community Health Systems Emergency Medicine Family Health Care Nursing HDF Comprehensive Cancer Center Inst for Global Health & Global Health Sciences Physiological Nursing Preventive & Restorative Dental Sciences Proctor Foundation Radiation Oncology Urology

Summary of Phase 1A Six-month Evaluation Results 5

CLIENT SATISFACTION RESULTS 6

Client Satisfaction- Faculty satisfaction with pre-award services BEFORE RMS implementation AFTER RMS implementation Phase 1A Faculty only Pre-award Baseline Questionnaire Summer '11 (114 respondents; 48% response rate) Notes: *High satisfaction responses= "Strongly agree"/"Agree" OR "Excellent"/"Satisfactory" **Neutral satisfaction responses= "Neither agree nor disagree" OR "Inconsistent" ***Low satisfaction responses= "Strongly disagree"/"Disagree" OR "Unacceptable" ++ This data represents a quick quality improvement effort to uncover early indicators of Phase 1A client experience. The theoretical margin of error is 12.5%, plus or minus, 95% of the time. Phase 1A How's My Driving? quick survey ++ Faculty that submitted proposals, Oct '11- Feb '12 (42 respondents; 31% response rate) 7

Client Satisfaction Department Managers & Post-award staff 8 Positive feedback Overall, Department Managers and Post-award staff are optimistic about RMS working successfully, despite some initial challenges. The majority of Department Managers rated their satisfaction with RMS as neutral- neither satisfied nor dissatisfied. Majority of Post-award staff are overall satisfied with RMS. RSCs are good about clearly communicating timelines and instructions to PI. RSCs are responsive. RSCs are knowledgeable. Proposals seem to be reviewed more thoroughly by RMS. Department Managers like knowing that their RSC is supported by a team of peers and that they have a secondary "back-up" RSC.

Client Satisfaction Department Managers & Post-award staff Areas for improvement Action Items Workload: Some Department Managers perceive their RSC as "overworked.“ Continuing to monitor the RSC workload as well as the impact of individual departmental internal procedures on RSC workload. Inconsistency across RSCs' service delivery: RMS is highly accommodating and customer-service oriented, which creates difficulty in standardizing services across departments. Developing a consistent approach to roles and responsibilities across all RSCs through on-going training and refining the Service Level Agreement. Lack of proactivity in communications with PI: Some RSCs need to be more proactive in scheduling in-person meetings and moving the proposal development process along. RSCs to offer an in-person introductory meeting with all of their client faculty. RSC assignment changes were disruptive: Some departments received more than 1 RSC assignment over the six-month period. Developed a transition plan for departments if an RSC assignment change is necessary. Proposal review turn-around: When proposals are submitted early, RMS does not always comply with the 5-day turn-around time. When reviewing proposals, RMS requests last minute corrections with quick turn-around. Enforcing the policy of 5-day review turn-around regardless of sponsor due date. With only five days to review, last-minute corrections are necessary but minimized whenever possible. 9

QUALITY OF WORK RESULTS 10

Quality of work- Critical Errors During the Phase 1A six-month period:  0% proposals were rejected by the sponsor due to critical errors  100% proposals were submitted by the sponsor's deadline 11

INTERNAL PROCESSES RESULTS 12

Internal Processes- RMS staff satisfaction 13 Positive aspects of the job "Extraordinary" team environment: Colleagues to share expertise and to share workload. Peer-reviewing proposals is of great value to RSCs: improves proposal quality training for proposal reviewing enhances exposure to a variety of proposal types Increasing knowledge about research administration. Staff like feeling empowered to make changes and improve processes as the first team under OE. Staff feel more comfortable working with C&G than before RMS or when first started.

Internal Processes- RMS staff satisfaction Challenges of the job Action Items Concerns about finding the balance between customer service and enforcing standard service levels. Evaluate the SLA and continue to provide tools and training. Concerns about how the 10 RMS teams will interact and be standardized. Developing cross-team communication tools, trainings and Townhall meetings. Utilize RMS website and Salesforce chatter. Some RSCs are uncertain and/or concerned about their workload in the future and how much they can handle. Continuing to monitor the RSC workload as well as the impact of individual departmental internal procedures on RSC workload. Doing the "C&G pieces" has added to the complexity of the Team's work and may be, among others, a contributing factor in their workload concerns. During initial and on-going training provide more detailed procedures on the "C&G pieces." PI last minute proposals create a burden on the process and affect quality of work. The goal is to “do the best we can in the time we have”; make sure PIs know what might not be reviewed in a time crunch; work with PI on proposal timelines; work with Department Managers and Chairs to manage "repeat offenders;" RMS staff struggle with the lack of tools and the manual use of Excel to track and report out on activities. Juggling deadlines makes administrative and data entry "housekeeping" difficult for the RSCs. Currently developing systems to enable more automated generation of reports and reduce data entry. Departments that have the same RSC as prior to RMS implementation are having difficulty understanding the change in processes and roles of their RSC/former RSA. Create tailored materials targeted towards departments and their RSCs in this situation to highlight their specific changes. 14

Internal processes- Proposal volume Phase 1A Team Productivity TOTAL Phase 1A only (8/15/11- 2/14/12) Pre- OE Phase 1A Departments' Estimated Six-month Proposal Volume (Summer 2011 baseline data) Number of external proposals completed by Phase 1A Team Total Number of "Proposal Activities" (internal proposals plus sponsor correspondence) TOTAL Pre-award Matters completed by Phase 1A Team

Internal processes- Proposals per RSC * Metric derived from workload of 7 RSCs over 4.5 month period and 1 RSC over 3.5 month period during Phase 1A. ** Metric derived from 12 Phase 1A departments self-report in Summer Average annual proposals per RSC End state performance target annual proposals per RSC Phase 1A*Baseline** Average The Phase 1A Team's productivity rate exceeded the average productivity rate of the pre-OE Phase 1A Departments. As the pilot group, Phase 1A Team was not expected to reach targeted proposal volume due to the complexity and challenge of rolling out the new initiative.

Internal processes- Accountability 17 MeasureResultsAction Items Monthly Activity Reports The Monthly Activity Reports are submitted to Department Managers to inform them of the current pre-award activity in their unit. Reports were sent out for all months, however not within the first week of the following month and in one case a month late. Department Managers want monthly activity reports consistently submitted on time. Post-award staff want more timely status updates on pending sponsor correspondence. RMS staff note that the monthly activity report is largely a manual process that requires 2 workdays to complete for all team clients. Continue to track submission of timely monthly reports. Continue to work on developing automated reports to reduce staff time devoted to this activity. Currently developing systems to enable more automated generation of the monthly activity report.

COST RESULTS 18

Costs- Average cost per proposal Phase 1A + Summer 2011 BaselineVariance $2,418* $2,523** $(105) Phase 1A baseline summer 2011 min and max: $ 864** $ 6,053** * Used actual FY12 fringe benefit rates. ** Escalated actual FY11 fringe benefits by 4%. + Does not include Transition Services effort and proposals. 19

TRAINING PROGRAM EVALUATION RESULTS 20

Training program evaluation results 21 Overall strongly positive reaction to the training program immediately following both the Phase 1A and 1B programs. After Phase 1A, no average change in knowledge and a slight decrease in confidence was found. Program revisions were implemented in Phase 1B resulting in an increase in knowledge and confidence. Phase 1A manager and participants felt the team-building modules were effective. However, some areas of the curriculum were difficult to apply on-the-job because they were too theoretical. Continuous improvement efforts are succeeding as noted in the increased learning outcomes of Phase 1B. RMS staff report that follow-up and one-on-one trainings are very helpful and applicable. These training opportunities continue to develop.

Implementation Challenges During the first six-months of implementation, a number of challenges impacted the RMS Phase 1A team and OE Implementation Team capacity including: Increasing and unexpected requests from departments for pre-award support before their scheduled roll-in to RMS created a high volume of unexpected workload. Accelerating the implementation schedule with the development of Phase 1B resulted in less focus on Ph1A needs and decreased the capacity to carefully plan/prepare for future phase roll outs. Identifying and preparing space was more time consuming than anticipated and affected capacity to focus fully on the implementation of teams. Recruitment continues to be time-consuming, and affected timely implementation of Phase 1B. Accommodating unique internal processes of departments proves to be difficult while trying to achieve efficiency goals. 22

Implementation Keys to Success Strong senior leadership support "High touch" and structured change management Adapted approach as needed during implementation of RMS Size and composition of OE Pre-award Implementation Team enabled constructive guidance and timely decision-making Faculty advisement on evaluation and monitoring led by Claire Brindis, DPH 23

Implications and Future Direction Communication with faculty and staff clients will be an on- going priority. – An "RMS Client Listserv" is being developed for broad distribution of news and information. – A new, comprehensive Office of Sponsored Research website is currently under-development to effectively provide information for the clients of RMS and the Contracts & Grants Specialty Unit. – The quarterly Research Administration Town Halls will continue. Addressing the personalized training needs of the RMS staff will continue through the ongoing training sessions. Hired two full-time Training Specialists to develop and continuously improve training opportunities for RMS. 24

Implications and Future Direction- cont'd After only six months, it's premature to draw conclusions about staff workload levels. Close monitoring of RSC workload and satisfaction will continue. Factors affecting workload are: – RMS staff's overall steep learning curve – working in a new service center organizational structure – working with a new team of colleagues and supervisor – working with new PIs, Department Managers and post-award analysts – learning new systems and new delegated institutional authority processes, procedures and policies Costs of the RMS are directly related to workload. During the implementation phase, over-staffing was purposely planned to manage the steep learning curve and allow for model refinements. Our Phase 1A objective of no increase in average cost per proposal was achieved. Once steady state workload targets are achieved, cost targets should be achieved. 25

Acknowledgments With special thanks to the following for their invaluable contributions: Department Managers, Faculty and Staff of Phase 1A departments for their hard work and willingness to make Phase 1A a success Phase 1A team for their hard work and dedication to the success of RMS Dean Hawgood and the OE Faculty Oversight Committee OE Pre-award Implementation Team Research Administration Board (RAB) SoM Dean's Office and PMO staff dedicated to managing the project 26

APPENDIX: METHODS 27

Client Satisfaction- ‘ How's My Driving?’ Phase 1A Faculty Quick Survey Methods Methods: Survey distributed 5 times over six-month evaluation period and around major proposal deadlines from Oct '11- Feb '12. Sent only to faculty that recently submitted a proposal through RMS Utilization-focused: To determine what went well and what needs improvement Brief survey delivered to faculty clients Four Likert scale questions about service delivery satisfaction and 2 open-ended questions request sent from the RMS Director, Marge O'Halloran Reasons for changing the survey methods at the six-month mark: Do not want to over-survey faculty – Do not want to burden faculty with more questions and survey questionnaires than needed Targeted survey efforts to Phase 1A faculty who used RMS services by inviting them to complete the How's My Driving quick survey – After 4 months, only about 75 of the 240 Phase 1A faculty used RMS services. Of the 75, only 32 responded to the original Summer '11 survey 28

Client satisfaction Department Managers & Post-award Staff Methods Shortly following the six-month mark, two focus groups were conducted-- one with Department Managers and another with Post-award Staff. – At the focus group sessions, 5-question likert-scale satisfaction surveys were distributed. – Phase 1A staff unable to attend the focus group were invited to complete an open-ended questionnaire. Of 11 Phase 1A Department Managers: – 6 attended the focus group session – 2 completed a questionnaire Of 12 Phase 1A Post-award Staff: – 7 attended the focus group session – 1 completed a questionnaire 29

Methods cont'd 30 MeasureData Collection Method Evaluative Criteria # of proposals rejected by sponsor due to critical errors Team Manager and Contracts & Grants records and Team Manager interview. Acceptable quality= 0% Unacceptable quality= more than 0% # of proposals submitted by sponsor's deadline Team Manager and Contracts & Grants records and Team Manager interview. Acceptable quality= 0% Unacceptable quality= more than 0% RMS staff satisfactionFocus groups, one-on-one interviews, short Likert scale surveys Exploratory, inductive content analysis of interview responses Proposals per RSC compared to baseline Proposal Express and Phase 1A's Proposal Log and Summer 2011 Baseline data provided by Phase 1A Departments Compare progress to Phase 1A Departments' pre-OE average Accountability- Monthly Activity Reports Count shared drive report records and interview RMS staff and Ph1A Dept Mgrs Reports submitted monthly to all departments= good less than monthly= inadequate Cost per proposalProposal Express, Proposal Log, Weblinks reports and financial records provided by the EVCP Office Compared to Ph1A departments' pre- implementation baseline cost per proposal range Training program evaluation Kirkpatrick design: reaction survey, learning test tool, progress report interviews with participants' manager Monitoring data