1. A Comprehensive Assessment Approach for the School of Advanced Studies Rob Olding, Ph.D. Associate Dean for Assessment University of Phoenix: School of Advanced Studies

2. Learning Assessment for Doctoral Level Education is …? Relatively new in current conceptualization Relatively abstract (compared to Associate and Bachelor levels) Higher thresholds for expectations of existing skills Research and writing intensive

3. A Multiple Method Approach Assessment Strategy ­ Somewhat in the form of an “Instrumental Case Study” approach consisting of “data vectors” from various perspectives: ­ Rubric based assessment based on IRMA mapping of curriculum ­ External evaluation sources (e.g., accreditor reports, Council on Graduate Schools, external consultants). ­ Appreciative Inquiry approach to assessment

4. Triangulation of Assessment Data Sources (basic model) SAS Practitioner Doctorate Learning Assessment Data Sources Rubric based measurement: Student Learning Outcomes External 3 rd Party Data (Reports) Appreciative Inquiry (AI) Based Assessment Deliberative Analysis

5. Rubric based measurement: Student Learning Outcomes ­ Critical to the assessment process is the development of analytic rubrics to measure performance on signature assignments as described. ­ As documented in multiple sources, analytic rubrics have been shown as effective feedback even at the education level (Groggins-Selke, 2013) ­ As documented in multiple sources, analytic rubrics have been shown as effective feedback even at the advanced education level (Groggins-Selke, 2013)  Analytic Rubrics focused upon:  Signature assignments in content and research courses  Key Progression assignments such as dissertation concept paper and proposal  Doctoral Dissertation assessment via multiple reviewers as “capstone” measure of doctoral success

6. External 3 rd Party Data (Reports) ­ Leveraging the data from these different sources to form a clear analytical picture of how our learning outcomes and programs are perceived from external views.  External Consultant Reports as engaged by the University of Phoenix School of Advanced Studies – (e.g., Council of Graduate Schools)  Survey Data (e.g., SEOCS and Alumni Surveys)  Accreditor Reports (e.g., Higher Learning Commission)

7. Appreciative Inquiry approach to Learning Assessment ­ This approach is argued to be atypical in assessment of learning, but very appropriate for the Doctoral level which compared to lower level degrees can be diverse, abstract and dynamic in outcomes.  As a method is known to be “transformational” and “engaging” which is important for both faculty and students  Positive in focus: “What were the most important learning outcomes you found in this course”?  Tied directly to a process of envisioning what we do best, what our dream outcomes would be, and how we might move to obtain that level!

8. Appreciative Inquiry Steps: 4-D Model adapted from Watkins and Mohr (2001) Discovery “Best of what is?” Appreciating Dream “What might be?” Envisioning Results Design “What should be the ideal? ” Destiny “How to empower, learn, improve?” Sustaining Change

9. ­ The three approaches to assessment data provide a diverse set of perspectives ­ These perspectives can be left unreconciled, or… ­ Integrated to provide a new level of perspective that informs far beyond any on its own. ­ TRIANGULATION Integration of Multiple Data Sources using diverse methods

10. ­ Includes diverse perspectives in the discussion ­ Provides for a basis of meaningful engagement for administration, faculty and students ­ Avoids an over reliance on a single paradigm approach that limits perceptions Advantages:

11. ­ The Triangulation process requires a means of bringing together diverse data sources ­ An ideal approach would extend the process to multiple “stakeholders” that would include students, faculty and administration ­ Such a method would work to ensure “fair consideration” of each data source in relation to optimum learning outcomes Means of Integration:

12. The Deliberation Model An inclusive approach that involves all stakeholders and represents each of the data sources in the process

13. Deliberation: A means of integrating diverse perspectives ­ Deliberation is neither a casual discussion or a formal or informal debate. The discussion should focus first on each source individually and then in combination and from a “best solution” view  Maximum engagement – Can be used in face to face, online synchronous and asynchronous formats  Allows for carefully weighing diverse views and bringing light upon points not previously considered and explored  Allows for exploration of facts in relationship to strategic positions as represented by each area data source

14. External Reports (3 rd party) Rubric based Assessment Appreciative Inquiry Results Interpretation and Integration Each perspective is sequentially discussed in terms of meaning and both positive and negative implications. Then they are compared and discussed in relationship to integration with each other. Commonalities are of particular note in TRIANGULATING meaning.

15. ­ The strategy is to: ­ Maximize engagement of stakeholders ­ Optimize the data sources in ensuring that diverse methods and perspectives are included in consideration ­ Develop an integrated set of insights that can inform strategy and contribute meaningfully to continuous improvement ­ Move assessment and evaluation to a new level for Doctoral Education Insights and Outcomes:

16. Thank You! Questions????

17. REFERENCES Borkowski, Nancy, A. (2006). Changing our thinking about assessment at the doctoral level. In Maki, Peggy L. & Borkowski, Nancy A. (Eds.), The assessment of doctoral education: Emerging criteria and new models for improving outcomes (pp. 11-51). Stylus Publishing LLC, Sterling, Virginia. Bushe, Gervase R. & Kassam, Aniq F. (2005). When is appreciative inquiry transformative?: A meta- case analysis. The Journal of Applied Behavioral Science, Vol. 41, No. 2, pp. 161-181. DOI: 10.1177/0021886304270337. Groggins Selke, Mary J. (2013). Rubric assessment goes to college: Objective, comprehensive evaluation of student work. Rowman & Littlefield Education, Lanham, Maryland.