1. Recontextualizing Bloom’s Taxonomy: Quantitative Measures in Formative Curriculum Assessments and Program Evaluations presented by Anthony Clemons Student, Master of Education Teachers College, Columbia University Aaron Smith, Ph.D. Lecturer, Dept. of Mathematics University of Central Florida Presented March 12, 2016 28th International Conference on Technology in Collegiate Mathematics

2. AGENDA Purpose of Our Research Supporting Literature Modified Delphi Assessment Model Conclusion References

3. RESEARCH PURPOSE We wanted to find a way to:  Formatively evaluate the quality of an academic course through its curriculum,  Using cognitive achievement as the measurement vehicle,  While quantifying our results

4. RESEARCH PURPOSE What would this evaluation require?  A “reasonable surrogate” (Horner et al, 2005 )  That assimilates assessment data  To provide a comprehensive evaluation of both a course and its curriculum

5. RESEARCH PURPOSE How did we arrive at a finding a “reasonable surrogate?

6. SUPPORTING LITERATURE Bloom’s Taxonomy Quantitative Model Reasonable Surrogate This evaluation is possible using,

7. SUPPORTING LITERATURE What is Bloom’s Taxonomy?  A hierarchal scheme for identifying a continuum of six specific cognitive demands in learning, ranging from simple (knowledge, comprehension, and application) to more complex (analysis, synthesis, and evaluation) (Bloom et al.,1956; Armbruster & Ostertag, 1989).

8. SUPPORTING LITERATURE Higher-Order Thinking Skills (HOTS)?  The taxonomy’s top three cognitive domains  Requires students to grasp a deep understanding of what they are learning and be more critical and creative instead of merely recalling information

9. SUPPORTING LITERATURE Lower-Order Thinking Skills (LOTS)?  The taxonomy’s lowest three cognitive domains  Emphasis on instruction versus facilitation  Involves “repeated responses to recurrent stimuli” for the purpose of adjusting cognitive tendencies so individual knowledge might be thwarted to action (Dewey, 1916).

10. SUPPORTING LITERATURE Why Use Bloom’s Taxonomy?  Research concedes it as a valid framework for quantitatively measuring the consistency and quality of HOTS and LOTS, it can also be an input for categorizing and comparing cognitive skills  It defines the academic quality of courses “in terms of the level of academic demands or rigor expected of the students” (Nordvall & Braxton, 1996, p. 486)

11. MODIFIED DELPHI ASSESSMENT MODEL How do we use Bloom’s as a formative assessment tool? NOTE: Nature of formative curriculum assessments is that they are conducted prior to curriculum being implemented as an instructional instrument (i.e. before the coursework is taught in class). By modifying Rowe and White’s (1999) characterization of the classical Delphi method, this need can be met through redefining the method’s key features via an iterative Delphi-based model.

12. Tri-Phased Delphi Evaluation Process for Programmatic Curriculum Assessments MODIFIED DELPHI ASSESSMENT MODEL

13. Delphi 1: Coding Learning Objectives  Learning objectives are assessed for validity and coded according to the taxonomical domain with which each objective is aligned as follows: Taxonomical LevelTaxonomical Code Knowledge1 Comprehension2 Application3 Analysis4 Synthesis5 Evaluation6

14. MODIFIED DELPHI ASSESSMENT MODEL Delphi 1: Coding Learning Objectives  Using table below, synchronizing the taxonomical domain coding for a course’s learning objectives can be facilitated:

15. MODIFIED DELPHI ASSESSMENT MODEL Quantitative Model Delphi 2: Quantitative Curriculum Assessments Statistical MeasurePurpose Arithmetic MeanEstimate of the sampled population’s central tendency Weighted Distributive Mean Ensures the percentage of time spent in a class session on each learning objective is accurately captured using a weighted percentage VarianceProvides a quantitative basis for consideration as to whether to narrow or widen the variance in how curriculum and learning objectives foster learning

16. MODIFIED DELPHI ASSESSMENT MODEL Quantitative Model Delphi 2: Quantitative Curriculum Assessments Statistical Measure Purpose Standard DeviationProvides a quantitative basis for consideration as to whether to narrow or widen the variance in how curriculum and learning objectives foster learning. One-to-one relationship with the variance. Used according to preference MedianAllows for researchers and practitioners to determine the quality of distribution through knowing what the measure for central tendency is for all of the learning outcomes within a course. SkewnessMeasures whether there are observable extreme values following a common direction along either the first and fourth quartiles, incurring an asymmetric form to the central tendency along that direction.

17. MODIFIED DELPHI ASSESSMENT MODEL What are we looking for? Cognitive Progression and an emphasis on HOTS!

18. MODIFIED DELPHI ASSESSMENT MODEL Quantitative Model Delphi 3: Content Analysis, Content Revisions, Learning Objectives Recoded, and a Quantitative Reassessment  Lessons endure a another review for learning objectives and lesson content alignment  Some lessons must be recomposed by:  Adjusting the lesson content to meet the actual cognitive level of the objective,  Adjusting the lesson learning objective(s) to the taxonomically sufficient level that meets the actual cognitive level of the content, or  Removing the objective and content from consideration entirely

19. MODIFIED DELPHI ASSESSMENT MODEL Delphi 3: Content Analysis, Content Revisions, Learning Objectives Recoded, and a Quantitative Reassessment  Once all appropriate revisions successfully instituted, the actions within Delphi’s 1 and 2 must be re- engaged  This ensures the most accurate data is accumulated giving all stakeholders a decisively precise snapshot of the state of a course’s curriculum Achieved through using this model and Bloom’s taxonomy as lesson assessment and course evaluation vehicles =

20. CONCLUSION This model allows institutions to have a structured, measurable body of curriculum for its courses, while leaving the content creativity to the instructor and department This model provides a quantitative assessment methodology for accreditation agencies to use in determining the level of “critical thinking” being achieved in every course, department, and institution This can provide consumer data to prospective students who want to effectively achieve HOTS in their coursework We acknowledge the need for additional research, using various data sets, but the existence of course learning outcomes and lesson learning objectives must exist to do so

21. REFERENCES Bloom, B., Engleheart, M., Furst, E., Hill, W., & Krathwohl, D. (1956). Taxonomy of Educational outcomes: The Classification of Educational Goals. Handbook I: Cognitive domain. New York: Longman. Dewey, J. (1916). Democracy and education: An introduction to the philosophy of education. New York: Macmillan Horner, R., Zavodska, A., & Rushing, J. (2005). How challenging? Using Bloom's Taxonomy to assess learning outcomes in a degree completion program. Journal of College Teaching & Learning, 2(3), 47-51. http://dx.doi.org/10.19030/tlc.v2i3.1785 http://dx.doi.org/10.19030/tlc.v2i3.1785 Rowe, G. & Wright, G. (1999). The Delphi technique as a forecasting tool: Issues and analysis. International Journal of Forecasting, 15(4), 353-375