Kristen St. John James Madison University stjohnke@jmu.edu Is there a better model for P&T preparation and evaluation of goescience education research (GER) in geoscience departments? Kristen St. John James Madison University stjohnke@jmu.edu Be sure to address both preparation (from a candidates perspective) and evaluation (from geo dept perpsective)

Motivation & Context: Personal Experience St. John and Leckie, http://serc.carleton.edu/NAGTWorkshops/earlycareer/tenure/preparing.html St. John (pseudonym Jones; 2007), http://chronicle.com/article/Tenured-Twice/46472/ my personal interest/background/motivation – I’ve given a lot of thought about tenure and promotion. I’ve serve on P&T committees at JMU, served as an external evaluator for P&T packages at other universities, received questions on what counts?, and with Mark Leckie offered Cutting Edge workshops on preparing for the tenure process, and I’ve even reflected on my own persona exp of getting tenure 2x (using a pseudonym).

Motivation & Context: Publication Trends Journal of Geoscience Education publication statistics Motivation & Context: Publication Trends # articles Year # pages Change in pub. standards/types of articles Trend over time – growth overall, plus shift to more evidence-based research (and away from here’s a neat thing I did in my class). Shift in format (online) and expectations in the last 10-15 yrs esp The impressions gained during this review were quite similar to those documented by Perkins (2004). …..JGE has primarily been a publication through which college and university geoscientists share ideas for creative teaching methods, …there appear[s] to have been a noticeable increase in the frequency of research articles within the last 10 years. There is also an increase in both the JGE and the wider literature in research papers on the subject of cognitive issues in teaching and learning. Data compiled by Erica Zweifel …there appear[s] to have been a noticeable increase in the frequency of research articles within the last 10 years. There is also an increase in both the JGE and the wider literature in research papers on the subject of cognitive issues in teaching and learning. – Pilburn, van der Hoeven Kraft, and Pacheco, (2011), A New Century for Geoscience Education Research. A commissioned study to inform the 2012 DBER Report.

Geoscience Education Research (GER): Graduate Programs by Institution Motivation & Context: Formal DBER Training Opportunities Geoscience Education Research (GER): Graduate Programs by Institution Program MS/PhD College Approximate timeline* Arizona State Science 1990 Mississippi State 2007 Michigan State MS/MACT/PhD Science & Education 1979 North Carolina State 2005 Northern Colorado MA Purdue Education Texas A&M 1992 University of South Florida 1971 Western Michigan Western Washington MS 2000 University of Northern Iowa MS, MA University of Wyoming 2009 University of Nebraska 2011 Northern Illinois 2013 Motivation and Context: Geo Ed Research Formal Training Compared to traditional research in geology, research on teaching and learning in the geosciences is a relatively young field of study. Peer-review publication statistics  and the increase in the number of graduate  programs in geoscience education research (GER) are evidence that this field is growing and robust.   Data compiled by Karen McNeal; primary source: Libarkin, https://www.msu.edu/~libarkin/about_programs.html *based on approximate hire dates of lead faculty

More Geoscience Education Tenure-Track Positions Motivation & Context: More Geoscience Education Tenure-Track Positions Most GER is conducted by faculty who are in tenure-track positions at colleges and universities. Therefore, career success requires a geoscience education researcher to plan for, and satisfactorily meet, the expectation for promotion and tenure  (P&T). P&T is traditionally organized around Teaching, Research, and Service . However, when the area of research is on teaching itself, where does this fit within the traditional P&T framework? And, how can P&T preparation and evaluation be improved?

Shared concerns about P&T evaluation by DBER journal editors Motivation & Context: Shared concerns about P&T evaluation by DBER journal editors Drafting Commentary for CHE or Change or Science or Nature…. Erin Dolan, Editor-in-Chief, CBE-Life Sciences Education Samantha Elliott, Editor-in-Chief, Journal of Microbiology and Biology Education Doug Everett, Editor-in-Chief, Advances in Physiology Education Charles Henderson, Senior Editor, Physical Review Special Topics - Physics Education Research Phillip Ortiz, Editor-in-Chief, Biochemistry and Molecular Biology Education Kristen St. John, Editor-in-Chief, Journal of Geoscience Education Marcy Towns, Associate Editor, Journal of Chemical Education

X Motivation & Context: Educational Psychology Geoscience Teaching Practice X Source: Lukes et al., JGE, 2015 Motivation & Context: Potential for GER marginalization/isolation in the geosciences. - Lukes et al., 2015; Feig 2013; and DBER report 2012, see esp. Dave Mogk’s summary http://serc.carleton.edu/NAGTWorkshops/DBER.html Real and time-sensitive issue for tenure-track geoscience education researchers in traditional geoscience programs. P&T evaluation of GER faculty is largely conducted by non-GER geoscience faculty and administrators. So X is part of Geosci, but not “mainstream” and the geosci persons evaluating P&T are probably mostly not situated at the X, they need to understand its value, and what metrics to use that can demonstrate excellence. Geoscientists have seen the boundaries between traditional geology and other science fields become more permeable as geoscience research questions have evolved to better capture the transdisciplinary nature of the natural world. From geochemistry to geophysics to geobiology, the value of a transdisciplinary approach to complex and meaningful research questions is evident in each of these fields. Research on geoscience teaching and learning is following a similar path, but has the added complexity of crossing over with the social sciences (e.g., educational psychology), which introduces research methods useful, but less familiar, to many geoscientists. P&T is traditionally organized around Teaching, Research, and Service. However, when the area of research is on teaching itself, where does this fit within the traditional P&T framework? And, how can P&T preparation and evaluation be improved?  each is a distinct category in which a faculty member must demonstrate excellence

P&T and GER Challenges: Appreciating the scope and methods of geoscience education research Doberneck et al., 2012. Beyond Activity, Place, and Partner: How Publicly Engaged Scholarship Varies by Intensity of Activity and Degree of Engagement, Journal of Community Engagement and Scholarship, v. 4. Publicly Engaged Scholarship Feig and Stokes, 2011, GSA Special Paper v. 474. Conditions that cause disconnects between GER and traditional P&T evaluation are outlined based on published studies and personal observation s. These include challenges that arise from integrating geoscience with social science ; from evolving definitions of what constitutes geoscience education research and scholarship ; and from different ideas on how the impact of research is best measured. Research methods today in GER are more rigorous than in past undocumented assertions not OK. Methods generate data! GER research is data-driven. http://jces.ua.edu/beyond-activity-place-and-partner-how-publicly-engaged-scholarship-varies-by-intensity-of-activity-and-degree-of-engagement/ Publicly engaged scholarship is often described by activity (e.g., service-learning; community-based, participatory research; public humanities), by place (e.g., rural communities, urban neighborhood), or by partner (e.g., non-governmental organization, school). These common descriptors—based on what faculty do, where they do it, and with whom they partner—fail to characterize how faculty members collaborate with community partners in engaged research, engaged teaching, and engaged service. This study explored whether two process-oriented constructs—level of activity and degree of engagement—were useful descriptors of how faculty members go about their scholarly collaborations with the public. Interpretive content analysis of 173 promotion and tenure forms revealed significant differences in intensity of activity and degree of engagement by gender, race, age, teaching assignment, joint departmental appointment, appointment length, Extension appointment, and discipline. These variations suggested new directions in professional development for community engagement and appointments/assignments supportive of faculty involvement in publicly engaged scholarship. The data in Table 3 indicate, for both intensity of activity and degree of engagement, that faculty members with primary appointments in education, health and medical professions, and agriculture and natural resources reported higher levels, while faculty members with primary appointments in business, arts and humanities, and physical and biological sciences reported lower levels. Faculty members in physical and biological sciences, the arts and mathematics, engineering, and computer sciences reported that service-learning (one type of publicly engaged scholarship) is “not relevant to their disciplines” (Abes, Jackson, & Jones, 2002, p. 12). In Figure 3 it is clear that the college grouping shows a positive relationship between intensity of activity and degree of engagement, with faculty in some colleges reporting both low intensities of activity and low degrees of engagement while others reported both high intensities of activity and high levels of engagement.

P&T and GER Challenges: Measuring Impact Thomson Reuters Science Citation Index (SCI) selected 2014 Journal Impact Factors The impact factor of an academic journal is a measure reflecting the average number of citations to recent articles published in that journal. Different playing fields can lead to misleading comparisons (e.g., expectation of external citations for DBER fields). Not all journals are indexed in Thomson Reuters (none in geoscience education), thus no SCI-impact factor. The impact factor (IF) of an academic journal is a measure reflecting the average number of citations to recent articles published in that journal. It is frequently used as a proxy for the relative importance of a journal within its field, with journals with higher impact factors deemed to be more important than those with lower ones.  Thank Bruce Herbert for helping me understand Impact factor better See Julie Libarkin’s Blog on Journal Impact Factors, and references therein: https://geocognitionresearchlaboratory.wordpress.com/2014/08/16/citations-and-impact-who-says-your-research-is-valuable/

P&T and GER Challenges: Measuring Impact Geoscience education research results Reform of geoscience education teaching and student learning institutional level national level One of the more important goals of DBER is translating research into practice. -DBER report, 2012. P&T Deliverable  programmatic reform of teaching/student learning. –Feig, 2013 This is a P&T deliverable that does not have parallels in basic traditional geoscience research. How can this impact be demonstrated? one of the more important deliverables for DBER that goes beyond publications is reform of program teaching/student learning is important – it is a deliverable that does not have parallels in basic traditional geosci research (non geo ed res).

Emphasize high quality of DBER [GER] work P&T and GER Recommendations: From the Community Emphasize high quality of DBER [GER] work -DBER report 2012, Mogk’s summary http://serc.carleton.edu/NAGTWorkshops/DBER.html Clarify expectation for DBER [GER] faculty positions Targeted marketing to non-GER colleagues – Feig, 2013 Be an active part of a GER community of practice – Lukes et al, 2015 To me these points sum-up in two ways: improve communication (within GER and between GER and geo communities) and define metrics of GER success. Emphasize high quality of DBER work  ensure that conclusions are evidence based (relates to morning session), methods are appropriate, and strengths and limitations transparent. Communication reduces isolation/marignalization. A synthesis is provided of recommendations from published reports on ways in which the challenges facing geoscience education researchers can be reconciled with the existing P&T structure. Recommendations range from community building  strategies to clarifying what constitutes evidence of impact for GER outcomes . Like in other geoscience fields, publications and grant-funding are key pieces of geoscience education research evidence, but there are nuances that need to be considered. For example, there is a wide range of dissemination outlets for geoscience education scholarship. These include: (1) publishing externally-reviewed DBER (e.g., JGE) and education-wide (e.g., JRST) research articles, (2) developing original, tested curriculum materials, and instructional or assessment resources accessible via open access (e.g., SERC) or commercial avenues, and (3) leading professional development workshops on geoscience teaching and learning (e.g., Cutting Edge Program). A P&T committee in a traditional geoscience program might view peer-reviewed publications as the only dissemination outlet of value. However, I think the reality is that effective translation of geoscience education research results into practice will depend on all three forms of dissemination. Therefore, all three forms should count in some way when evaluating geoscience education research for promotion and tenure. To what extent they count will depend on university P&T expectations, and thoughtful alignment of research agendas to community needs; it is therefore a necessary part of the local to national conversation that needs to take place on metric of success. Improve communication & define metrics of GER success

P&T and GER Recommendations: Additional Suggestions Tenure-track GER Faculty Talk with department colleagues early and often about what you do, how you do it, and why it is important - St. John and Leckie, 2009, http://serc.carleton.edu/NAGTWorkshops/earlycareer/tenure/preparing.html Avoid jargon. Maintain respect for colleagues. wisdom on geoscience teaching and learning isn’t exclusive to those with geoscience education PhDs. Find common ground. Geoscience itself. University mission, program goals P&T Committees Make informed decisions on what “counts”. Go beyond Impact Factors (e.g., download statistics) How to measure “translation of research into practice”?

Boyer (1990, Scholarship Reconsidered) P&T and GER Recommendations: Evolving the P&T Preparation and Evaluation Model Traditional Boyer (1990, Scholarship Reconsidered) Teaching Teaching Research (Discovery) Service (Application) Integration of Knowledge Research a modified structure  Boyer (1990) model of scholarship. In it, the porous nature of Teaching, Research, and Service is recognized and embraced by adding a forth P&T category of Integration. Integration of knowledge  “making connections across disciplines, placing specialties in a larger connect, illuminating data in a revealing way, often to educate non-specialists too.” Service  esp tied to ones field of specialty   Service Integration of knowledge  “making connections across disciplines, placing specialties in a larger connect, illuminating data in a revealing way, often to educate non-specialists too.” http://www.schev.edu/AdminFaculty/OFA/2016OFAGuidelines.pdf

(modified Boyer) Approach P&T and GER Recommendations: Evolving the P&T Preparation and Evaluation Model Metrics for Success An Integrated (modified Boyer) Approach Use GER best practices Evidence-based teaching/ model “translating research into practice.”) Impact on student learning Teaching Service (Application) Research (Discovery) Integration of Knowledge Research results should advance the GER field Publications Grant success In Career Planning In P&T Narrative In P&T Evaluation Criteria A deliberately integrated approach to career planning (P&T prep) and to P&T narratives, and P&T evaluation Meaningful connections across teaching, research (and service) would be expected in this structure; and geoscience education research, as well as other interdisciplinary fields of study, would be better situated for sharing the importance of their contributions.    In all cases, it will likely take both formal and cultural change in traditional geoscience programs in order to fully recognize the value of geo ed research – the change is not all the burden of the P&T committee – it is of the individual as well to demonstrate excellence, relevance, and impact.   Apply expert GER knowledge to department (e.g., curriculum com.) institution (e.g., Center for Teaching) GER community (e.g., NAGT GER Division) Geo community (e.g., Broaden Impacts on geo. grants, lead professional development workshops)