1. JCDL 2008 Exploring Educational Standard Alignment: In Search of ‘Relevance’ René Reitsma*, Byron Marshall* Michael Dalton*, Martha Cyr *Oregon State University Worcester Polytechnic Institute

2. JCDL 2008 Exploring Educational Standard Alignment: In Search of ‘Relevance’ Problem: aligning DL learning objects with educational standards –Need & tantalizing promise –National Science Digital Library (NSDL) efforts & accomplishments –Early results show low Inter-Rater Reliability (IRR) Hypothesis: low IRR is partially a methodological artifact Proposal: multifactor concept of ‘alignment’ Experiment: 10-factor alignment model –High IRR –Four factor regression model (R=.75) of ‘overall’ alignment

3. Aligning DL Learning Objects with Educational Standards Expanding DL learning resource base; e.g., –National Science Digital Library (NSDL): 928 collections –K-12: TeachEngineering.org, TeachersDomain.org, Engineering is Elementary, etc. –NSF-GK-12 program (ongoing). ≈84,500 math, science & technology standards (changing frequently) JCDL 2008

4. Curriculum Standard Alignment Efforts NSDL leadership: –Jes&Co: Achievement Standards Network (ASN) –Center for Natural Language Processing (CNLP): Curriculum Alignment Tool (CAT) Standard Alignment Tool (SAT) –WGBH Teachers’ Domain: standard alignment & lexicon Others: –Academic Benchmarks –AAAS/NSDL Strandmap server –Etc. JCDL 2008

5. NSDL-based Curriculum Alignment Services JCDL 2008

6. How Good are these Alignments? JCDL 2008

7. Low Inter-Rater Reliability (IRR) Devaul, H., Diekema, A.R., Ostwald, J. (2007) Bar-Ilan, J. Keenoy, K., Yaari, E., Levene, M. (2007) “There is no average user, and even if the users have the same basic knowledge of a topic, they evaluate information in their own context…” Hypothesis: Low IRR is partially a methodological artifact –Alignment is a multifactor, multidimensional concept –Learning objects may align with certain dimensions but not with others –Levins, R, Lewontin, R.C. (1980): “Abstraction becomes destructive when the abstract becomes reified… so that the abstract descriptions are taken for descriptions of the actual objects” JCDL 2008

8. Dimensions of Alignment Seracevic, T. (2007): ‘ Relevance: A Review of the Literature and a Framework for Thinking on the Notion in Information Science. Part II: Nature and Manifestations of Relevance ’ JCDL 2008 “Clues”Our(!) Mapping to Educational DLs ContentTopics & concepts ObjectCost, learning object type, formatting ValidityTrustworthiness Use/Situational matchGrade level, institutional requirements; e.g., testing procedures, professional development Cognitive matchTeacher qualifications, pedagogy Belief /Affective matchEmotional response

9. Hypotheses H: One-dimensional alignment/relevance IRR is partially a methodological artifact. H-1: At least some dimensional IRRs will be high(er) H-2: Dimensional IRR will vary H-3: ‘Overall alignment/relevance’ IRR will be low, even when asked in the context of dimensional relevance testing. JCDL 2008

10. Experiment ‘Clue’AlignmentStatement Affective matchR-1 AppealThe document contains materials that are motivational or stimulating (interesting, appealing or engaging) for students ContentR-2 ConceptsThe document includes concepts, keywords, terms and definitions from the standard ContentR-3 BackgroundThe document provides interesting and important background material related to the standard ObjectR-4 Grade levelThe grade level of this material is appropriate for this task or else I can easily adapt the materials in this document to my grade level Situational matchR-5 NontextualsI can use (a) nontextual component(s) ; e.g., figures, tables, images, videos or graphics Situational matchR-6 ExamplesI can use the real-world examples provided in the document in class. Situational matchR-7 Hands-onI can use one or more of the hands-on, active engineering activities Situational matchR-8 AttachmentsI can use some of the attachments; e.g., score sheets, rubrics, test questions, etc. Situational matchR-9 ReferencesI can use references or Internet links to relevant materials elsewhere R-10 overall relevanceOverall, I consider this document relevant for this teaching assignment JCDL 2008

11. Experiment Cont.’d 14 Subjects all familiar with the TeachEngineering system Two teaching tasks: –“As a third grade Massachusetts teacher you are assigned to teach material related to the standard “Relate earthquakes, volcanic activity, mountain building, and tectonic uplift to plate movements.” You have two hours of class time to spend on instruction.” Judge the alignment of three curricular objects (R-1 – R-10, six-point Likert scale) JCDL 2008 IRR-1: both subjects score on the same side of the scale; i.e., both score either ‘strongly agree,’ ‘agree,’ or ‘somewhat agree,’ or both score ‘somewhat disagree,’ ‘disagree,’ or ‘strongly disagree.’ IRR-2: same as IRR-1 except that answers may not differ with more than one scale point. IRR-3: both subjects answer the question identically

12. Results 91 IRR comparisons × 10 alignment dimensions × 6 alignments H-1: IRRs are relatively high; IRR-1 (binary): 64%-95% H-2: IRR variability H-3: Overall relevance (R-10) among the weaker ones JCDL 2008

13. How about Overall Relevance (R-10)? r R-1R-2R-3R-4R-5R-6R-7R-8R-9R-10 R-1 Appeal 1.0 R-2 Concepts.161.0 R-3 Background.36.631.0 R-4 Grade.27.20.311.0 R-5 Nontextuals.09.19.20.181.0 R-6 Examples.21.26.27.30.421.0 R-7 Hands-on.34.31.40.59.24.401.0 R-8 Attachments -.02.17.21.36.35.42.391.0 R-9 References.06.23.20.30.25.24.45.301.0 R-10 Overall.31.76.66.50.28.35.54.35.341.0 JCDL 2008

14. MLR Model of Overall Relevance (R-10) JCDL 2008 R 2 =.75 ‘Overall alignment/relevance’ is meaningful as a complex variable. Some high IRR alignment dimensions do not contribute to overall alignment. βStd. ErrorT-valuep Intercept-.272.259-1.048.298 R-2 Concepts.567.0836.794<.01 R-3 Background.173.0842.047.044 R-4 Grade level.322.0903.576<.01 R-7 Hands on.194.0932.082.041

15. Conclusion JCDL 2008 K-12 educational DL content is expanding; educational standard alignment is needed. Innovative and promising resources are available but reported IRR of assessment of alignments is low. Propose that ‘Alignment’ is a complex concept: –Recognize alignment dimensions –Experiment suggests that dimension-specific IRR will be (much) higher –‘Overall’ alignment has a very specific interpretation. What do we need: –Continued assessment and IRR collection –Collections making their assessment data available. –Alignment methods that can assimilate ‘evidence’ from the multiple dimensions that comprise ‘alignment’ of a learning resource with a teaching standard.