Network Visualization of Human and Machine-based Educational Standard Assignment IV10 – London, UK René Reitsma 1, Anne Diekema 2 1 Oregon State University 2 Utah State University

Network Visualization of Human and Machine-based Educational Standard Assignments – Overview Problem: need for automated educational standard assignment. Question 1: how are machine-based assignments (e.g., CAT) different from human (cataloger) assignments? Question 2: how do humans assign these standards in the first place? (Inductive) method: –Generate standard networks of both the human and machine-based assignments. –Explore the structure and visual layout of both networks for indications of differences.

The Standard Assignment Challenge DL CollectionK-12 itemsEstimation method Teachers’ Domain ( 3,373Search query on TeachEngineering ( 812Document listing on Website NetTrekker ( 300,000Promotional video Engineering is Elementary ( 78Direct count on Website Middle School Portal ( 2,562Search on Curriki ( 32,222Resource count displayed on home page National Science Digital Library ( 79,155Estimate by NSDL official

The Standard Assignment Challenge… cont.’d In USA, K-12 teaching & learning is increasingly standards based (‘No Child Left Behind’ Act – 2001) Educational standards (K-12) are governed by the 50 states. States have different views and opinions  50,000+ K-12 Science, Technology, Engineering & Math (STEM) standards: MT: 20, AK: 50, IA: 60,..., NY: 2,309, IL: 2,881, TN: 3,410 Teachers must find curriculum to teach to the standards. Curriculum providers must assign standards to their curriculum in order to assist the teachers. Available curriculum grows quickly; states regularly change/update their standards.

The Standard Assignment Challenge… cont.’d Manual assignment is (very) time consuming  Automatic assignment tools have been under development for some time. Tool of choice: Content Assignment Tool (CAT): –Center for Natural Language Processing (CNLP) – Syracuse University –Part of NSF’s National Science Digital Library ( –NLP technology – part-of-speech tags (verbs, nouns, etc.), TF/IDF, etc. –CAT web service: Point CAT to an XML/HTML version of a curricular document. Tell it which standard body to assign from. Returns a list of assigned standards.

That’s All Very Nice, but… How good are those CAT assignments? What do we really mean when we say that a standard and a curricular item line up? (Reitsma, Marshall, Zarske (IPM – 2010)) If the human and CAT assignments are different, are these differences systematic? Do the human assignments provide suggestions on how to improve the machine-based ones?

(Inductive) Method & Data Notion: visualize both machine-based and human assignments and compare and contrast the visualizations for clues. Approach: build networks of standards; layout the networks, interpret their spatial arrangements: –Networks are based on how standards have been assigned to curriculum. –Any two jointly assigned standards are considered ‘linked.’ Data: –TeachEngineering collection ( – Jan –CO 2007 Science standards. –CAT & human standard assignments.

CO 2007 Science Standard Assignments Human Catalogers (CO Curriculum) CAT (CO curriculum cataloged by humans) Curricular items assigned 86 Assignments Standards covered 6347 Standard reuse rate Mean number of curricular items per standard Mean number of assignments per curricular item

CO 2007 Science Standard Assignments... cont.’d CAT assignments (CO curriculum cataloged by humans) YesNoTotal Human cataloger assignments Yes No114NA Total139 – CAT recall = 25 / 324 =.077 – CAT precision = 25 / 139 =.18

‘Curricular units’ – Human network is denser and more clustered. – Human curricular units are clusters – Human clusters link through common standards. – CAT: open structure; less clustering. Has no sense of curricular units.

Weighted or unweighted? FR diagrams consider the network unweighted; i.e., all links have equal value/weight. Two weights: –TF/IDF-like: weigh a standard link inversely proportional to the size of its company. –‘Fidelity:’ weigh a link between standards proportional to their mutual fidelity across the collection. Compute the KK network layouts

Resulting KK diagrams showed essentially the same properties as the FR diagrams (hierarchical cluster analysis of two-dimensional positions)

Standards: ‘Method’ vs. ‘World’ CO Science standards can be grouped and subdivided in two categories: –World standards (W): express facts and principles about the empirical world. E.g., S103EC87: Light and sound waves have distinct properties: frequency, wavelength and amplitude. –Method standards (M!): express ways and means of conducting science. E.g., S103ECE9: A controlled experiment must have comparable results when repeated. –Some method standards ‘contaminated’ with world terms/examples (M): E.g., S103ECD4: Technology is needed to explore space (for example: telescopes, spectroscopes, spacecraft, life support systems). Question: How do CAT and human catalogers compare on World vs. Method?

Standards: ‘Method’ vs. ‘World…’ cont.’d StandardsM & M! standards M & M! standards % M! standards M! standards % Humans CAT

W = world M! = (pure) method M = method with world examples – CAT under-assigns method. – Humans: method standards as curricular hubs – CAT central method hub: S103EC77: “physical properties of solids, liquids, gases and the plasma state and their changes can be explained using the particulate nature of matter model“

Conclusions & Recommendations Visualization of standard assignment networks reveals important differences between CAT-based and human-based classification. Tools such as CAT can benefit from contextual knowledge; e.g., that certain lessons are part of a larger set of lessons or a curricular unit. Humans organize curriculum around both World and Method standards. Hence, tools such as CAT MUST!! become better at recognizing Method standards. In USA strong need for automated standard assignment. CAT is a recognized and well-used automated classifier. CAT could use improvement, particularly in the Method segment.