AAER Conference Nov Symposium on Limitations of Scientific Knowledge in Educational Research: Bias, Non-Significant Findings & Knowledge Representation Jaya Kannan Larry Lutsky Yasmine Alwan Lisa Bauer Non-Significant Findings and a Research Response in Self-Directed Learning

Analyzing students’ articulation of learning goals: How Problems in Research Led to Solutions in Teaching Jaya Kannan Larry Lutsky Yasmine Alwan Lisa Bauer

Purpose of the presentation Share research work on LGs Discuss rubric development Examine what didn’t work and what was learned Elicit feedback from participants

Brief History of Learning Goals Andragogy: theories of adult learning (Knowles) Learning goals individualize learning & contribute to the development of self-directed learning

The Purpose Centered Model Relevance of Learning Goals Constructive Action teaching practices Seeing connections in goal setting in different contexts

Method Hypothesis Sample Procedures Data collection Method of analysis Type of design Rubric

Evolution of the Rubric Need to design a rubric Method of working – cyclical Establishing inter-rater reliability Scale items Operational definitions Math and writing – learning contexts Examples for each scale

Evolution of the rubric Disagreements/clarification: Defining Specificity Defining complexity What constitutes measurability? Should we use a zero scale?

Evolution of the rubric Resolutions Defining Specificity – Break it up into content and syntax Defining complexity – Establishing parameters for what is/is not a multi- layer goal –case by case evaluation Should we use a zero scale? – Only for measurability What constitutes measurability? – Arrived at keywords

Rubric – early version 123 Specificity Not specific – I want to learn math Moderately specific – I want to understand mathematical concepts Highly specific – I want to learn basic algebra. I want to get an “ A ” in my course. Complexity Low complexity – I want to learn how to multiply and divide fractions. Moderate complexity – I want to learn about statistics so I can analyze the data from my CA Highly complex – I want to learn algebra so that I can apply the concepts in my course and also apply outside the class room. Measurability Low measurability – I want to have more confidence in my ability. Moderate measurability – I want to improve my math skills High measurability – I want to be able to solve quadratic equations. I want to be able to calculate the mean and standard deviation. I want to be able to determine if there is subject-verb agreement

Rubric – final version 1= Low23 = High Specificity – Content Specificity - Syntax I want to learn math. I want to understand mathematical concepts. Broad area superset I want to learn basic algebra Less broad subset I want to solve algebraic equations Narrow subset I want to understand fractions Describes an action To improve my math skills by completing math problems Describes an action in detail To learn sign numbers so I can solve equations correctly Describes more than one action with detail ComplexityI want to learn fractions 1 layer of goal I want to learn how to multiply and divide fractions >=2 layers of goal I want to learn about statistics so I can analyze data from my CA >= 2 layers of goal with application MeasurabilityI want to improve my math skills Any desirable change Stated demonstration of ability I want to reduce the numbers of errors I make while solving math equations by the end of the semester Something quantifiable By the end of the semester, I want to reduce the number of errors I make and complete more work independently. 2 layers - Quantifiable, time frame

Results of data collected

Overall change in means over semestersChange in first 2 means, divided by # of sessions

Results of data collected

Flaws/Challenges in Methodology Applying hypothesis to problematic data Retroactive analysis Abundant data Edu purposes Lack of standardization Goals Raters Measurement Definitions of scale items Consensus reached through extensive discussion

What was learned Standardization is difficult to achieve in teaching practice Should it be standardized in the first place? Limits in student knowledge and skills Actual work vs. projected work Teaching styles

Follow up: Next Steps Discussions with specialists improve understanding (using the rubric) uniformity Revising goal writing form Building in measurability Generating templates of goals

Follow up: Next Steps Collaboration of LEC and faculty Participant feedback on rubric & study Connection between research and teaching