We Need Your Help What we need you to do for us: If we build it, will you use it? Be willing to test-drive the user interface and provide feedback. Help seed the database with assessments and data from your course(s) What is the FIRST Database? Faculty Institutes for Reforming Science Teaching (FIRST) has engaged faculty from over 50 institutions in professional development focused on active, inquiry-based teaching designed to improve student learning. We are constructing a database to support research on undergraduate STEM education. The database will support storing, searching and analyzing assessment data from undergraduate STEM courses. This database will facilitate both data-driven instructional decision making (Figure 1) and research in science education. FIRST Assessment Database: Your Role Database Functionality We are using a large variety of assessment data collected at Michigan State University and from colleges and universities across the nation to design the FIRST Assessment Database. Faculty from all STEM disciplines will input and retrieve data from the database to explore questions about effective teaching and learning in undergraduate education (Figure 2). This project will facilitate cross-institutional studies using assessment data from large numbers of students and classes. The database is the bridge between teaching and research that enables faculty to become both expert users of and contributors to the scholarship of scientific teaching. Your Research Question Did teaching students to create, interpret and critique models result in better student understanding of evolution? Figure 1. Two models of instructional design. Metadata - data about data To promote STEM education research, you tag assessment items with metadata, such as Discipline-specific concepts. In biology, for example, we plan to use the National Biological Information Infrastructure (NBII) Biocomplexity Thesaurus (Figure 5). Additional metadata tags will include standard psychometrics such as difficulty and discrimination, Bloom’s Taxonomy of Educational Objectives, Professional society tags and whether an assessment item is copyrighted (as in a textbook question or a published concept inventory). Metadata collected by the FIRST database will also encompass information about courses and institutions (Figure 6). These data will facilitate longitudinal studies in addition to comparisons among courses across institutions. Figure 5: Example of concepts retrieved from the NBII Biocomplexity Thesaurus ( Figure 3. Data sources from a faculty member, including: (a) output from course management software, (b) excel spreadsheets of student exam, homework and in- class assignments, (c) clicker data, (d) student open-ended responses and (e) exams. Your Data Sources You upload to the FIRST Database all assessments and related data from your introductory biology course (Figure 3), including assessment questions and grading rubrics. You tag each assessment item with both a concept category (see Figure 5) and an instructional strategy (i.e., modeling, pair-share, JiTT, etc). You have the option to tag assessment in several additional ways (see the discussion of Metadata, next column), or you may bypass this task at this time. Security, Intellectual Property, IRB Institutional review boards (IRB) are unique to each college and university. This reality demands a flexible database that is both secure and protective of student data. The FIRST Assessment Database will support two levels of data access - restricted (available only to the faculty member who uploaded the data) and public (available to all registered database users). Many assessment items are copyrighted - as in textbook questions or published concept inventories. Faculty members using the database must be aware of and take responsibility for obtaining appropriate permissions for copyrighted or otherwise protected material. Without such permissions, an assessment item cannot be part of the public database. For more information on copyright issues and intellectual property rights, please see our website. Student responses to the FIRST Assessment Database are de-identified by a hashing function that ensures student anonymity. To promote longitudinal studies, however, the database will link students across courses at an institution. A faculty member may chose or be directed by their institution’s IRB to remove identifying information about themselves and the institution. This material is based upon work supported by the National Science Foundation (NSF) under award Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF. Figure 2. Overview of the FIRST Assessment Database. Figure 4. Potential data output from the FIRST database. Data Analysis To analyze whether teaching students to make models improved their learning of evolution, you query your current course and two past courses for data on evolution and modeling. The FIRST database returns a single spreadsheet file with the data from each course (Figure 4). You import the data into your favorite statistical analysis software. As a correlated question, your are interested in comparing your students’ modeling abilities with those of students from other institutions. Your again search the FIRST database for courses at other colleges or universities using model-based education to teach evolution. You limit your query to include only large introductory biology courses at institutions similar in size to yours. You then limit the returned data to include only assessments related to modeling in evolution. The FIRST database again returns a spreadsheet that you can import into statistical software for analysis. Based on your analysis, model-based learning led to significant gains in student learning of evolution in your course and courses at other universities. These results prompt you to redesign additional portions of your course to include modeling. Institution type/size Course format (lab, lecture, discussion, etc) Course size Targeted students (majors, non-majors, lower or upper level) Course Syllabus Type of assessment (e.g., in-class, open book, exam) Proportion of final grade Bloom’s level of understanding Concept category Course-level Metadata Assessment-level Metadata Figure 6: Examples of course- and assessment-level metadata captured by the FIRST Assessment Database. Data-driven instructional design Analyze student learning outcomes Analysis of student data drives course modification Design course Teach course Assess student outcomes Content-driven instructional design Discipline-based knowledge (textbooks) For more information, please contact Diane Ebert-May or Mark Urban-Lurain Design course Teach course Assess student outcomes Discipline-based knowledge (textbooks) Formative feedback Selection Evolution Genetic drift Group selection Host selection Kin selection Natural selection Adaptations (biological) Ecophenes Fluctuating asymmetry Genetics Mutation Phylogeny Selective media Speciation Competition Diversity indices Dominant species Environmental effects Fitness Genetic load Genetics Survival value