Using Concept Maps for Evaluation Combines results from 2 projects conducted across 2 years: Using Concept Maps Transcribed from Interviews to Quantify the Structure of Preschool Children’s Knowledge About Plants (Wehry, S., Algina, J., Hunter, J., & Monroe-Ossi, H.) & Healthy Habits Through Literacy: A Concept Mapping Curriculum for Preschool and Prekindergarten Children (Monroe-Ossi, H., Wehry, S., Algina, J., & Hunter, J.) The Florida Institute of Education at the University of North Florida

Using Concept Maps for Evaluation  A shared goal of the Young Florida Naturalists & Healthy Habits Through Literacy projects – examine concept mapping as a tool to quantify conceptual development in preschool children.  Researchers need a reliable and valid way to use children’s concept maps for program evaluation.

Scoring System: 3 Components Propositions 0 points if irrelevant0 points if irrelevant 1 point if an example of concept1 point if an example of concept 2 points if describes an attribute of concept2 points if describes an attribute of concept 3 points if states a purpose of the attribute3 points if states a purpose of the attribute Cross-Links 5 points each5 points each Hierarchical Levels 5 points each5 points each

Scoring System: Examples

Interrater Reliability Study  Data: Young Florida Naturalists Assessment 2 Interviews (n = 48).  Maps: 1 researcher mapped all interviews.  Raters: 3 trained researchers rated all maps.  Analysis: Estimated how much variance was accounted for by raters, maps, and raters*maps.

Interrater Reliability Study: Results

Concept Mapping Protocol  The 3 Young Florida Naturalist raters used the 48 interviews to develop a mapping protocol. Propositions do not have to be grammatically correct.Propositions do not have to be grammatically correct. Do not map irrelevant propositions.Do not map irrelevant propositions. Mapped propositions can stand alone, that is, not connected to the focal concept.Mapped propositions can stand alone, that is, not connected to the focal concept. When in doubt, the benefit goes to the child.When in doubt, the benefit goes to the child. Score the resulting concept maps to test ease of scoring.Score the resulting concept maps to test ease of scoring.

Interrater & Intermapper Reliability Study  Data: Healthy Habits Through Literacy Assessment 2 Interviews (n = 48).  Maps: 3 researchers mapped all interviews.  Raters: 3 trained researchers rated all maps.  Analysis: Estimated how much variance was accounted for by the interviews, mappers, raters, and all interactions of the 3 sources.

Interrater & Intermapper Reliability: Results

Validity Young Florida Naturalist - Prekindergarten children were assessed in the fall and spring (n = 26). The Test of Early Reading Ability -Third EditionThe Test of Early Reading Ability -Third Edition (TERA-3) (TERA-3) Bracken Basic Concept Scale -Third Edition: Receptive (BBCS-3:R)Bracken Basic Concept Scale -Third Edition: Receptive (BBCS-3:R) Average Concept Map ScoresAverage Concept Map Scores

Validity Study: Results Correlations with Concept Map Scores Note: *p <.10; ** p <.05; SRC is the School Readiness Composite.

Validity: Regression Results After controlling for gender, age, prekindergarten class, and pretest score, the children’s concept map scores predicted spring: BBSC - 3:R Texture/Material scale achievement.BBSC - 3:R Texture/Material scale achievement. BBSC - 3:R Self-/Social Awareness scale achievement.BBSC - 3:R Self-/Social Awareness scale achievement.

Validity 2: Measures Healthy Habits Through Literature -Prekindergarten and preschool children were assessed in the spring (n = 35) using the following measures: The Peabody Picture Vocabulary Test - Fourth EditionThe Peabody Picture Vocabulary Test - Fourth Edition (PPVT-4). (PPVT-4). The Expressive Vocabulary Test – Second EditionThe Expressive Vocabulary Test – Second Edition (EVT-2). (EVT-2). Bracken Basic Concept Scale -Third Edition: ReceptiveBracken Basic Concept Scale -Third Edition: Receptive (BBCS-3:R). (BBCS-3:R). Bracken Basic Concept Scale -Third Edition: ExpressiveBracken Basic Concept Scale -Third Edition: Expressive (BBCS-3:E). (BBCS-3:E). Average Concept Map Scores.Average Concept Map Scores.

Validity 2: Results Correlations with Concept Map Scores Note: *p <.10; ** p <.05; SRC is the School Readiness Composite

Conclusions  Concept maps developed from preschool children’s transcribed interviews can be reliably scored.  Preschool children’s transcribed interviews can be reliably mapped and scored.  Resulting concept map scores correlated with measures of expressive vocabulary, attributes of people & objects, and alphabet knowledge.

Discussion Revise Scoring System 1. To reflect the use of Dynamic vs. Static1. To reflect the use of Dynamic vs. Static Statements. Statements. 2. Concerning the use of examples.2. Concerning the use of examples. Revise Mapping Protocol Revise Mapping Protocol 1. To reduce variance due to mapper in cross-1. To reduce variance due to mapper in cross- link & hierarchical level scores. link & hierarchical level scores. 2. To better fit across curricula.2. To better fit across curricula.

Discussion Why rate concept maps rather the interviews? Rating the interview did not provide clarity in determining the structure of children’s knowledge relative to the domain (Novak & Musondo, 1991)Rating the interview did not provide clarity in determining the structure of children’s knowledge relative to the domain (Novak & Musondo, 1991) Concept mapping children’s interviews makes visible the their propositions, cross-links, and hierarchical structure.Concept mapping children’s interviews makes visible the their propositions, cross-links, and hierarchical structure. The concept maps make visible places where instruction can be improved.The concept maps make visible places where instruction can be improved.

Next Steps Professional Development: 22 prekindergarten teachers to use concept maps to facilitate children’s learning.22 prekindergarten teachers to use concept maps to facilitate children’s learning. Kindergarten teachers to use concept maps to help children connect their background knowledge to newly acquired information.Kindergarten teachers to use concept maps to help children connect their background knowledge to newly acquired information. Middle School Implementation: Use concept mapping strategies with 6 th and 7 th graders in after-school programs at two middle schools.Use concept mapping strategies with 6 th and 7 th graders in after-school programs at two middle schools.