Lauren Weiner, Ed.D. Director, Associated Students Administration University of California-San Diego Marilee J. Bresciani, Ph.D. Professor, Postsecondary.

Slides:



Advertisements
Similar presentations
The PRR: Linking Assessment, Planning & Budgeting PRR Workshop – April 4, 2013 Barbara Samuel Loftus, Ph.D. Misericordia University.
Advertisements

USING NEEDS ASSESSMENTS & CREATING ACTION PLANS September 25, 2012 Greg Lobdell, Center for Educational Effectiveness Candace Gratama, The BERC Group Travis.
Assessment Cycle or Circular File: Do Academic Librarians Use Information Literacy Assessment Data? Megan Oakleaf & Lisa Hinchliffe Library Assessment.
Town Hall Presentation January 9-10, 2002 Curtis Powell Vice President for Human Resources The Division of Human Resources and William M. Mercer, Incorporated.
2009 Strategic Planning playbook
360 Degrees: Conducting a Comprehensive Evaluation of Your Integrated Planning Processes Bri Hays Jill Baker San Diego Mesa College RP Conference April.
Student Affairs Assessment Workshop Presented by: Dr. Gary R. Hanson Educational Leadership & Policy Studies Arizona State University Sponsored by: NASPA.
1 Department of State Program Evaluation Policy Overview Spring 2013.
Identifying Your Role in Outcomes-Based Assessment Program Review Marilee J. Bresciani, Ph.D. Professor, Postsecondary Education and Co-Director of the.
1 Phase III: Planning Action Developing Improvement Plans.
Charting a Sustainable Future: Marshall University Vision 20/20 Campus Conversation October 6, 2014 Service Portfolio Review.
Bresciani, M.J. Implementing Lessons Learned: Establishing A Culture of Evidence- Based Assessment Marilee J. Bresciani, Ph.D. Associate Professor, Postsecondary.
Strategic Budgeting Initiative Provost Open Forum November 19, 2013.
NCCEA Annual Conference Waynesville, NC Assessment Basics: Implementing and Sustaining a Comprehensive, Outcomes-Based Assessment Plan October 19, 2006.
The Pennsylvania State University Service Learning- Engaged Scholarship Task Force Sponsored by Interim Provost Vice President, Student Affairs Vice President,
A DISSERTATION PROPOSAL PRESENTED BY: CARTER FILE Understanding the Impact Career Paths have on Community College Chief Business Officers’ Employment Satisfaction.
A brief overview What is program evaluation? How is an evaluation conducted? When should it be used? When can it be used? Used with Permission of: John.
Institutional Effectiveness Operational Update Presentation made to the Indiana State University Board of Trustees October 5, 2001.
Are you Using Technology to Assess Learning or Assessing Learning to Improve Technology? Marilee J. Bresciani, Ph.D. Associate Professor, Postsecondary.
Marilee J. Bresciani, Ph.D. Professor, Postsecondary Education and Co-Director of the Center for Educational Leadership, Innovation, and Policy San Diego.
Title I Needs Assessment and Program Evaluation
HELPFUL TIPS FOR UNIT PLANNING Office of Institutional Effectiveness.
Student Services Assessment Lee Gordon Assistant Vice President for Student Services Purdue University.
RESEARCH DESIGN.
Maria Thompson Provost & Vice President for Academic Affairs Academic Affairs Restructuring Proposal 23 April 2012.
Mia Alexander-Snow, PhD Director, Office for Planning and Institutional Effectiveness Program Review Orientation 1.
ASSESSMENT  Are you having an impact?  How do you know?  Are the programs and services you work on getting better?  How do you know?
Exploring Good Practices in Outcomes-Based Assessment Program Review Marilee J. Bresciani, Ph.D. Associate Professor, Postsecondary Education and Co-Director.
Performance Management Open Information Session for Individual Contributors.
University Strategic Resource Planning Council Budget.
May Priority 6 Review Team 6: Planning Subcommittee R. Hall-Allen, A. Cohen, D. Cohen, M. DesVignes, M. Hong, D. Kinney, S. Maradian, J. Ireland,
NASPA Presentation Practical Tools for Building Division-wide Assessment Capacity Adrienne Dumpe, Graduate Assistant, VPSA Katie O’Dair, Director of Assessment.
Connecting Course Evaluations to Program Evaluations Marilee J. Bresciani, Ph.D. Associate Professor, Postsecondary Education and Co-Director of the Center.
Performance Management  Performance Management Cycle  Organizational Success  Shared Responsibilities  Setting Goals and Expectations.
Program Review In Student Affairs Office of the Vice President Division of Student Affairs Virginia Tech
Successful Contract Training: A Grounded Theory for a Sustainable Business Model presented at the National Council for Workforce Education Conference by.
WHAT DO WE KNOW ABOUT THE QUALITY OF HIGHER LEARNING IN THE UNITED STATES? Marilee J. Bresciani, Ph.D. Professor of Postsecondary Education Leadership.
A project of the President’s Emerging Leaders Program in collaboration with the Administrative Services Task Force.
HECSE Quality Indicators for Leadership Preparation.
San Antonio College SAC SCORES: PROGRAM ASSESSMENT Presented By: Dr. Kris Clark Cynthia Price, M.A Dr. Franc Solis.
STUDENT SERVICES REVIEW January 8, Context – Administrative Unit Reviews Objectives Roles Unit Self-Study Internal Review Committee External Reviewers.
Don Dodson, Senior Vice Provost Diane Jonte-Pace, Vice Provost for Undergraduate Studies Carol Ann Gittens, Director, Office of Assessment Learning Assessment.
University Planning: Strategic Communication in Times of Change Cathy A. Fleuriet Ana Lisa Garza Texas State University-San Marcos Presented at the July.
University of Idaho Successful External Program Review Archie George, Director Institutional Research and Assessment Jane Baillargeon, Assistant Director.
Marilee J. Bresciani, Ph.D. Professor, Postsecondary Education and Co-Director of the Center for Educational Leadership, Innovation, and Policy San Diego.
What could we learn from learning outcomes assessment programs in the U.S public research universities? Samuel S. Peng Center for Educational Research.
Office of the State Minister of Reforms Coordination of Georgia /GORBI Monitoring Governance Reforms Pilot Survey in Tbilisi Schools.
Meeting the ‘Great Divide’: Establishing a Unified Culture for Planning and Assessment Cathy A. Fleuriet Ana Lisa Garza Presented at the 2006 Conference.
The NCATE Journey Kate Steffens St. Cloud State University AACTE/NCATE Orientation - Spring 2008.
Budget Presentation December 7, Why Talk about the Budget? 1. Demystify the process 2. Engage and get the participation of others in the budget.
By Monica Y. Peters, Ph.D. Coordinator of Institutional Effectiveness/QEP Office of Quality Enhancement.
Learning Goals at St. John Fisher College Peter J. Gray, Ph.D. Director of Academic Assessment United States Naval Academy May 2004.
Marilee J. Bresciani, Ph.D. Professor, Postsecondary Education and Co-Director of the Center for Educational Leadership, Innovation, and Policy San Diego.
STRATEGIC PLANNING & WASC UPDATE Tom Bennett Presentation to Academic Senate February 1, 2006.
Periodic Review Report Committee Summary: Amended Monthly Progress Report December 22, 2003.
Afdeling Institusionele Navorsisng en Beplanninig Division for Institutional Research and Planning FROM EVALUATION REPORTS TO ACTION – CLOSING THE LOOP.
Indiana University Kokomo Strategic Enrollment Management Consultation Final Report Bob Bontrager December 8, 2007.
Session 2: Developing a Comprehensive M&E Work Plan.
HLC Criterion Five Primer Thursday, Nov. 5, :40 – 11:40 a.m. Event Center.
A Template for Stakeholder Dialogue. Purpose To move its students to greater success with college and career readiness and in other state priorities the.
Standard Two Les Steele Executive Vice President.
Help Me Understand the Basics of Non-academic Assessment A BETHUNE-COOKMAN UNIVERSITY OFFICE OF ASSESSMENT WORKSHOP Presented by Cory A. Potter Executive.
1 Establishing a New Gallaudet Program Review Process Pat Hulsebosch Office of Academic Quality CUE – 9/3/08: CGE – 9/16/08.
COD Institutional Effectiveness Process (IEP)
IT Governance Planning Overview
Linking Strategic Planning and Budgeting
Budget Presentation December 7, 2006.
Strategic Budgeting Initiative
Developing an information portal
Presentation transcript:

Lauren Weiner, Ed.D. Director, Associated Students Administration University of California-San Diego Marilee J. Bresciani, Ph.D. Professor, Postsecondary Educational Leadership San Diego State University Ben Gillig, M.A. Candidate Assistant Coordinator of Academic Initiatives San Diego State University Lisa McCully, M.A. Director, College of Education Office of Student Services San Diego State University Utilizing an Outcomes-Based Program Review to Inform Resource Allocations Bresciani, Gillig, Weiner, & McCully

Session Overview Presentation of framing questions Review of research study Discussion of findings and application to practice Presentation of a preliminary framework Discussion with small groups Bresciani, Gillig, Weiner, & McCully

Framing Questions What evidence is used to inform your resource re-allocation or allocation processes? How does your outcomes-based program review process inform resource re-allocation or allocation process? Why would you even want to have this conversation? Bresciani, Gillig, Weiner, & McCully

Background Resource allocation is about more than just money…  Human resources  Facilities  IT  Time spent by senior leaders  Budget allocations  Evidence that priorities are being effectively met Resource allocation and re-allocation are cornerstones of senior student affairs officers’ jobs How resources are allocated says something about how the division operates and how it sets its priorities Bresciani, Gillig, Weiner, & McCully

Student Affairs Leaders Pulling The Pieces Together Divisional Budget Allocation Human Resources Donations Capital Inputs Programs & Services Outputs Resource Allocation Bresciani, Gillig, Weiner, & McCully

Literature Review Budgeting Frameworks  Program budgeting  Formula budgeting  Zero-base budgeting  Performance budgeting  Incremental budgeting Budgeting Frameworks  Program budgeting  Formula budgeting  Zero-base budgeting  Performance budgeting  Incremental budgeting Require a set of purposes, priorities, and/or strategies to work. What framework do we use to ensure that our purposes, priorities, and/or strategies will be effective in accomplishing intended results? Bresciani, Gillig, Weiner, & McCully

Purpose of the Study This mixed method study explores how institutional/divisional leadership use outcomes-based assessment results to inform resource allocations and re- allocations for student learning and development. Explores the need to utilize outcomes-based assessment in times of budget cuts, reduction in revenues, and a higher demand for services The findings intend to identify frameworks that align resource allocations and re-allocations with stated priorities and provide evidence to refine programs and services that align with those priorities. Bresciani, Gillig, Weiner, & McCully

Hypotheses The manner in which institutions are funded does not influence how institutional leadership uses outcomes-based assessment results to inform resource allocations and re-allocations for student learning and development. The timeframe in which institutional leadership allocates and re-allocates resources for student learning and development does not influence how they use outcomes-based assessment results to inform the allocation and re-allocation of resources for student learning and development. Bresciani, Gillig, Weiner, & McCully

Hypotheses, Cont. The budgeting framework that institutional leadership uses to allocate and re-allocate resources for student learning and development does not influence how they use outcomes-based assessment results to inform the allocation and re-allocation of resources for student learning and development. The manner in which institutional leadership engages in strategic planning does not influence how they use outcomes-based assessment results to inform the allocation and re-allocation of resources for student learning and development. Bresciani, Gillig, Weiner, & McCully

Hypotheses, Cont. The manner in which institutional leadership engages in outcomes-based assessment program review does not influence how they use outcomes- based assessment results to inform the resource allocation and re-allocation of resources for student learning and development. Bresciani, Gillig, Weiner, & McCully

Methodology This mixed methods study included the following data collection processes: Survey list started with institutional members (n=1,030) Researchers stratified the institution by type A random sample of 257 invitations were then sent through Student Voice. Survey invitations were sent to the institution’s Senior Student Affairs Officer  E.g., VP of Student Affairs or Student Services Bresciani, Gillig, Weiner, & McCully

Methodology After data was collected, researchers analyzed the information using descriptive statistics Researchers looked for correlations between institutional/divisional budget process and how they utilized their outcomes-based assessment results for their resource allocations and re-allocations Researchers conducted a document analysis based on the information that was provided from their institutions URL addresses Interviews will be conducted in the upcoming weeks Bresciani, Gillig, Weiner, & McCully

Survey Participant Demographics Institution type Private 4-year Liberal Arts Colleges (35%) Public 4-year Comprehensive (20%) Public 2-year (15%) Institution size Less than 4,000 (37%) 4,000 to 8,000 students (24%) 15,001 to 25,000 students (22%) Position title Senior Student Affairs Officer (93%) Bresciani, Gillig, Weiner, & McCully

Instrument Development Survey included 25 questions and a request for institutions to provide their URL address for additional assessment documents Survey focused on asking institutional leaders to discuss if they utilize (a) outcomes-based assessment, (b) program review, (c) annual reports, and (d) strategic planning to inform the budget and resource allocation and re-allocation process at the institutional and divisional level Survey included questions pertaining to institutional/divisional leadership practices and questions relating to the practices used for allocating and re- allocating resources to student learning and development programs Bresciani, Gillig, Weiner, & McCully

Document Analysis Survey respondents were asked to provide their URL addresses Document analysis selection criteria: All materials needed to be official institutional documents, which included: Annual reporting processes Outcomes-based assessment processes Program review processes Assessment plans from the institutional, divisional, and departmental levels Researchers analyzed documents and developed themes by using the comparative method for category development Researchers determined essential descriptive groupings during the initial coding phase Bresciani, Gillig, Weiner, & McCully

Instrument Development Interview Protocol: Researchers will conduct semi-structured interviews with institutional leaders, such as vice-presidents, deans, faculty, and administrators directly involved in the outcomes-based assessment program review process Researchers will also conduct interviews with institutional leaders involved in the resource allocation and re-allocation processes Questions focus on: Types of evidence based decision-making processes that institutional leadership use to inform the resource allocation and re-allocation process for student learning and development The alignment between the results generated from outcomes-based program review and resource re-allocations and allocations Bresciani, Gillig, Weiner, & McCully

Limitations Since the researchers used a self-reporting survey, participants did not have to provide their actual budgeting documents, thus researchers cannot triangulate that data; however, outcomes-based and planning practices can be triangulated through document analysis 25% response rate; survey is back out in the field Bresciani, Gillig, Weiner, & McCully

Preliminary Findings Bresciani, Gillig, Weiner, & McCully

WE DO NOT HAVE CONCLUSIVE DATA FOR REJECTING THIS HYPOTHESIS The manner in which institutions are funded does not influence how institutional leadership uses outcomes- based assessment results to inform resource allocations and re-allocations for student learning and development. Bresciani, Gillig, Weiner, & McCully

Use of tuition and fee revenue was associated with use of OBA to formulate budget requests at both the institutional and division levels but not to inform actual resource allocation (t=-.43, p>.01; t=-.48, p>.01). Use of state allocated revenue was associated with OBA playing a role in re- allocation only (t=-.34, p=.02). Use of state allocation revenue was also associated with OBA playing a weighted role in the formulas used to allocate money to departments (t=.83, p=.01). Use of endowment revenue was associated with OBA playing a weighted role in the formulas used to allocate money to divisions (t=.31, p=.04). Use of revenue from auxiliary services was associated with OBA playing no role in resource allocation/re-allocation at all (t=-.32, p=.02). Based on these preliminary findings, we will be conducting interviews and gathering additional documents to clarify Bresciani, Gillig, Weiner, & McCully There were significant correlations between certain revenue sources and use of outcomes based assessment at certain points in the resource allocation/re- allocation process.

The timeframe in which institutional leadership allocates and re-allocates resources for student learning and development does not influence how they use outcomes-based assessment results to inform the allocation and re-allocation of resources for student learning and development. 98% of survey participants responded that their resource allocation process takes place on an annual basis Since the majority of respondents selected the same timeframe, we can not determine the extent that the time frame influences this process. Thus, we can not accept or reject this hypothesis Bresciani, Gillig, Weiner, & McCully

The budgeting framework that institutional leadership uses to allocate and re-allocate resources for student learning and development does not influence how they use outcomes-based assessment results to inform the allocation and re-allocation of resources for student learning and development. 46% report their institutions using incremental budgeting processes, while 35% use program- based at the divisional level There are no significant correlations between budgeting framework and the use of OBA for resource allocation and re-allocation We accept this hypothesis Bresciani, Gillig, Weiner, & McCully

The manner in which institutional leadership engages in strategic planning does not influence how they use outcomes-based assessment results to inform the allocation and re-allocation of resources for student learning and development. Strategic planning is the most common practice used to inform the resource allocation or re-allocation process Institution level: 22% of survey respondents reported that strategic planning informs the resource re-allocation process Institution level: 20% of survey respondents reported that strategic planning played a role in the way they construct the budget, but not the allocation of resources Division level: 22% of survey respondents reported that strategic planning informs the resource re-allocation process Division level: 22% of survey respondents also reported that strategic planning played a weighted role in the formulas used to allocate dollars There are no significant correlations between the use of strategic planning and the use of OBA for resource allocation and re-allocation We accept this hypothesis. Bresciani, Gillig, Weiner, & McCully

The manner in which institutional leadership engages in outcomes-based program review does not influence how they use outcomes-based assessment results to inform the resource allocation and re-allocation of resources for student learning and development. Institution level: 24% of survey respondents reported that program review does not have any role in the allocation or re-allocation of resources Institution level: 20% of survey respondents reported that program review plays a role in the re-allocation of resources Division level: 27% of survey respondents reported that program review plays a role in the re-allocation of resources Division level: 22% of survey respondents reported that program review does not play any role in the allocation or re-allocation of resources Bresciani, Gillig, Weiner, & McCully

The manner in which institutional leadership engages in outcomes-based program review does not influence how they use outcomes-based assessment results to inform the resource allocation and re-allocation of resources for student learning and development. However, uses of outcomes-based program review data was correlated with several uses of outcomes-based assessment data: Use of program review data in a weighted role in budget formulas used to allocate money to departments was correlated with the use of outcomes based assessment data for the same purpose (t=.43, p<.01). Use of program review data in a weighted role in the budget formula used to allocate money to programs and processes correlated with the use of outcomes based assessment data for the same purpose (t=.47, p<.01). Use of program review data to construct budget proposals, but not necessarily to allocate resources was correlated with the use of outcomes based assessment data for the same purpose (t=.67, p<.001). Use of program review data to construct budget proposals, but definitely not to allocate resources was correlated with the use of outcomes based assessment data for the same purpose (t=.69, p<.001). Thus, we cannot reject this hypothesis. Bresciani, Gillig, Weiner, & McCully

Preliminary Findings from the Document Analysis Documents analyzed from the URL addresses included annual reports, assessment plans, program reviews, strategic planning processes, assessment committee structures, and institutional planning documents Themes from the documents: Departmental goals and objectives align with the student affairs and institutional goals and objectives Institutions use assessment results to inform the program improvement process, not in the resource allocation process Identification of departmental and divisional successes, but do not inform the resource allocation process Bresciani, Gillig, Weiner, & McCully

Similarities and Differences Between Survey and Documents Institutional leaders identified and communicated values, objectives, and priorities Departments aligned programs with the institutional goals and objectives Input being solicited from faculty, staff, and administrators Strong participation of faculty, staff, and administrators in the planning process Differences between survey and documents: Survey respondents indicated that resource re-allocations were influenced by strategic planning and outcomes-based assessment program review findings; documents showed no linkage to the resource re-allocation process Bresciani, Gillig, Weiner, & McCully

Summary of Conclusions There is a disconnect between what revenue sources influence versus what the budgeting process influences. There is a disconnect between the institutional budgeting processes and the ability to use allocation of resources to improve student learning and development. Does that make sense to continue? If 74% of the initial allocation of funding is done annually, why aren’t outcomes-based program review results used to inform initial allocations of resources? Bresciani, Gillig, Weiner, & McCully

Summary of Conclusions, Cont. If strategic planning and outcomes-based assessment program review findings only influence resource re- allocations and not initial allocations, how will leadership be able to influence long-term improvement in student learning and development? More to come…. Bresciani, Gillig, Weiner, & McCully

Framework with Modifications (Bresciani, 2010 & Bresciani, et.al, in progress) Identify and articulate values Prioritize values Allocate resources Align outcomes to values and Implement outcomes- based assessment Define the criteria for quality within the context of the values and identify capacity for meeting the criteria of quality Gather the results and determine at which level the decision for resource re-allocation or allocation resides Bresciani, Gillig, Weiner, & McCully

Framework with Modifications (Bresciani, 2010 & Bresciani, et.al, in progress) Allocate or re-allocate resources to improve your outcomes within your context and capacity for quality and in alignment with your values Bresciani, Gillig, Weiner, & McCully

Small Group Discussion Questions How well do your resource budgeting and allocation processes align? How much do you utilize planning and assessment processes to allocate or re-allocate resources to refine your priorities? Does this proposed framework offer any practical value for your division/institution? What are some immediate next steps that you can implement? Bresciani, Gillig, Weiner, & McCully

Planned Next Steps for the Study Complete correlation analysis Re-invite participants to complete the survey Request more URLs for expanded analysis - particularly budget process documents Conduct interviews to test the viability of the framework Write-up results for publication submission Bresciani, Gillig, Weiner, & McCully

Special Thanks to… Student Voice  JD White  Annemieke Rice Participating institutions To you Bresciani, Gillig, Weiner, & McCully

Session Citation This presentation:  Weiner, L., Bresciani, M.J., Gillig, B., McCully, L. (2011). Using outcomes-based assessment to inform resource allocation in higher education. Workshop presented at the annual meeting of the American College Personnel Association, Baltimore, MD. Future paper:  Bresciani, M.J., Weiner, L., Gillig, B., McCully, L.(in preparation). Using outcomes-based assessment to inform resource allocation in higher education. Journal of Student Affairs Research and Practice submission Bresciani, Gillig, Weiner, & McCully

References Bresciani, M.J. (2010). Aligning values with resources and assessment results. Student affairs leader (38), 13, p Bresciani, M. J., Gardner, M. M., & Hickmott, J. (2009). Demonstrating student success in student affairs. Sterling, VA: Stylus Publishing. Berg, D.J. & Skogley, G.M. (1985). Editors’ notes. In D.J. Berg & G.M. Skogley (Eds.), Making the budget process work. New Directions for Higher Education No. 52. San Francisco: Jossey-Bass. Carnegie Council on Policy Studies in Higher Education (1980). Three thousand futures: The next twenty years for higher education. San Francisco: Jossey-Bass. Clark, R. & d’Ambrosio’s, M. (2006). The new balancing act in the business of higher education. Northamton, MA: Edward Elgar Books. Bresciani, Gillig, Weiner, & McCully

References, Cont. Merriam, S. B. (1998). Qualitative research and case study application in education. San Francisco: Jossey-Bass. Mayhew, L. (1979). Surviving the eighties: Strategies and priorities for solving fiscal and enrollment problems. San Francisco: Jossey-Bass. Strauss, A., & Corbin, J. (1990). Basics of qualitative research. Newbury Park, CA: Sage. Taylor, S. J., & Bogdan, R. (1998). Introduction to qualitative research methods. New York: John Wiley. Bresciani, Gillig, Weiner, & McCully

Contact Information Marilee J. Bresciani – Ben Gillig Lauren Weiner – Lisa McCully – Bresciani, Gillig, Weiner, & McCully