1 UST Support for Institutional Effectiveness: Information for Academic OA Plan Writers

2 Quality Enhancement Institutional Effectiveness defined  an ongoing, comprehensive, and institutionally integrated system  of planning and assessment  designed to enhance and improve the institution, as well as to demonstrate to what degree the institution and its components have been effective in fulfilling or achieving their stated mission or purpose.

3 Our Quality Enhancement/ Institutional Effectiveness activities include :  Strategic Planning Activities (intentional activity related to creating our “desired UST future,” based on planning, broad involvement, responsible agents, implementation, tracking, & continuous internal & external reviews)  Program Review Activities  Outcomes Assessment Activities

4 Outcomes Assessment Plans include:  Your Department’s Mission Statement  Conceptual Goals related to what you are trying to achieve as the legacies of your work (integrative & summative)  Measurable Objectives that permit you to shed light on the extent to which you are achieving these legacies  Assessment Plan indicating how you will measure objectives  Annual Summary of Findings  Annual statement of How Findings are Being Used to help the unit improve/enhance its efforts to achieve the legacies/outcomes

5 Your Mission Statement  Describes your mission in narrative.  Describes how your unit supports UST’s Mission/ Vision/ Strategic Agenda.

6 Conceptual Goals relate to Student Learning outcomes  Generate 3-5 student learning goals that support your mission and UST’S Mission/ Vision/ Strategic Agenda.  Articulate the integrative learning goals that your curriculum is designed to achieve.  Faculty own the curriculum; this is the process by which faculty define the outcomes your curriculum is designed to achieve in light of student learning.

7 Your Goals Drive Departmental Processes  Faculty should strive for consensus on student learning/ outcomes assessment goals.  Goals become a central foci for faculty activity.  Goals drive curriculum, teaching, and assessment.

8 Measurable Objectives permit you to shed light on the extent to which you are achieving your goals.  S imple: focus on a given goal.  M easurable: permit measurement.  A ffordable: can be done within budgets, with people in place.  R ealistic: are doable.  T imely: make sense in light of what you are doing.

9 SMART Objectives: S imple: Focus on a goal.  An objective should address one important function/aspect of a given conceptual goal…  Typically from one perspective.  You should have multiple objectives in order to provide multiple ways of shedding light on the extent to which a given goal is being reached.

10 SMART Objectives: M easurable: Permit measurement.  The objective should be written in measurable terms such that it acts as a standard or measuring stick by which you evaluate your efforts.  The objective is always focused on measuring an aspect of a given goal.

11 SMART Objectives A ffordable: Can be done within budgets, with people in place.  Most objectives should be something that you can manage within your current budget.  If new money is needed, consult your supervisors for testing affordability.

12 SMART Objectives R ealistic: Are “doable.”  Consider the all resources available to you including:  Financial  person-power  others  Set objectives that are ‘doable.’

13 SMART Objectives T imely: make sense in light of what you are doing.  The objectives should address issues of current importance such that the information obtained can be useful.  We need not expect to achieve and to measure all things all years.

14 An Assessment Plan for each objective  You need an Assessment Plan for each Objective.  The Assessment Plan measures what is identified in the related objective.  Briefly describe the assessment tool and/or procedure (what, who, how, when)  The assessment you include in the plan does not have to be something you have already been doing. It may be an assessment activity you will implement for the first time in in the OA Plan year.  Consult with colleagues, Dean, VPAA, with professionals in other institutions, with SPIRE for ideas, development of new tools, etc.

15 Types of Assessment Data  Tests  Surveys  Performance Evaluations (of students)  Scored Simulations  Portfolios that contain numerous pieces of evidence of student work and which are evaluated  Other evaluative measures

16 Tests as Assessments  Need to provide information of relevance to the faculty/curriculum  Locally developed vs. Nationally available  Local: time intensive vs. perceived appropriateness  National: lessened time commitment vs. relevance  Availability of reliable subtest scores, providing useful feedback.  Classroom exams, graded by a single faculty member, are not effective learning outcome measures of integrative learning.

17 Surveys as Assessments  Do not provide strong evidence of student learning; often used in OA Plans to assess service units.  Can focus on either perceptions or facts.  Can be completed by students-graduates or others (UG students’ graduate school professor, employers).  Do not measure learning directly.  Useful as adjunct for some purposes.

18 Performance Evaluations & Simulations  Require scoring rubrics to which faculty agree.  Evaluators must be trained in applying scoring rubrics.  High face validity, instructional validity, student and instructor acceptability.  Time intensive to take and score.  Multiple subscores preferred.  Necessitate reliability checks.  Scorers should not be all faculty; must have independence from teachers.

19 Portfolios  Assemblies of multiple types of info.  Replicate the same concerns as performance assessments & simulations  Need to have consensus, compilation, & summarization  Labor intensive, but good for looking at strengths and weaknesses, if multiple assessments are available

20 Evaluating Data Quality and Process Issues  Perceived validity = Feedback quality  Usefulness of resultant changes  Costs (time [student, faculty & staff], effort, and cost)  Impact on people in the department & the program  Sustainability  Justification of the program to prospective students and other constituencies  Acceptability to accrediting bodies

21 Frequently Utilized Means of Assessment For Administrative and Support Areas  Attitudinal measures of client satisfaction  Client/user survey  Direct measures or counts of area services  Volume of activity (number of persons served)  Level of efficiency (average time for responses)  Measure of quality (average errors per audit)  External Evaluation  Periodic assessment of the relationship of the department/unit’s efforts to “good and acceptable practice” by a “neutral” person who is knowledgeable in the field.  Student outcomes related measures (e.g., Student Satisfaction Survey findings)

22 Annual Summary of Findings  Be sure to report specific results of the listed objective/ assessments, including numbers where appropriate.  Provide a brief summary, including the numbers or specific examples of qualitative data found.  Maintain all support data and analyses for review, if requested.

23 Show How You Used Results to Improve Your Effectiveness, Teaching-Learning Interventions  Be sure to record program enhancement made as a result of findings.  Provide a brief summary of ways in which the results impacted your decision making, curriculum, teaching, other support behavior.  Include specific examples such as a change.

24 This is where we “Use Data to Close the Loop”  Data need to be evaluated by all  Data need to be quasi-public  Review on annual or more frequent basis  Serious time needs to be spent considering the findings—like research  Curricula are jointly owned; courses are parts of curricula. Instructional freedom is diminished by consensus.

25 OA Due Date: C ommunicated on Faculty Study Day 2003  Your Outcomes Assessment Plans Units OA Plans for are due to supervisors and VP (without results) by  OA Plans for , with findings and how they are being used to improve program effectiveness are due & for units with OA data not available earlier (e.g., Major Field Test results).

26 YOUR QUESTIONS? COMMENTS?