1. Undergraduates in the Archives: Measuring the Impact of Archival Instruction Through Rubric-based Assessment
Magia G. Krause
Ph.D. Candidate
School of Information
University of Michigan
Society of American Archivists Annual Meeting
August 14, 2009

2. Instruction and Assessment in Academic Archives
My dissertation looks broadly at the role of archivists as educators. It is comprised of three separate studies: (1) a survey about instructional practices, (2) interviews with archivists passionate about teaching, and (3) measuring learning from instruction.
This table comes from the survey of instructional practices I conducted last year.
Archivists devote a great deal of their time and effort on instruction, particularly for undergraduates. However, feedback about instruction is collected haphazardly and is often anecdotal so it is difficult to measure the impact of this instruction and know whether or not students are actually learning.
Krause, Magia G.(2008). Learning in the archives: A report on instructional practices, Journal of Archival Organization, 6(4)

3. Overview of Teaching Experiment
Research Questions
Do undergraduate students learn from archival instruction?
What do undergraduate students learn from archival instruction?
Goals of study
Introduction of a reliable assessment tool to measure student learning
Introduction of archival literacy scale that might compliment a curriculum of undergraduate research education in archives

4. Methodology Quasi-experiment in the field Background
US history course in a large state university
82 students total
Control and treatment groups
To compare students that did not receive archival instruction with students that did
Pre- and Post- Document Analysis Exercise
To provide a baseline comparison of the groups and look for an improvement in treatment group

5. Archival Instruction
Description of treatment (aka archival instruction)
Lecture/presentation to entire class by archivist
Hour-long session in the archives with 4 stations
Bibliographic Instruction
Critical Thinking
Photograph Analysis
Footnote/Citation Analysis

6. Photographs from the Instruction Session

7. Document Analysis Exercise
Divided into 3 sections
Adapted from NARA’s Teaching with Documents
Written Document Analysis
Photograph Analysis
Finding Aid Analysis
Difference between Pre- and Post- Tests
Alternative primary sources and finding aid

8. Assessing Student Learning
Rubric
Basic definition
Scoring guide that includes criteria and levels of performance
Types of rubrics
Holistic and analytic
Steps in creating a rubric
Criteria : Archival Literacy Skills
Observation
Interpretation/Historical Context
Evaluation/Critical Thinking
Research Skills
Gratch-Lindauer (2003) explains that designing a rubric typically requires a number of steps and decisions about the lesson content, levels of performance, and quality of work. She recommends following these steps: (1) describe the learning outcomes of the instruction; (2) identify specific attributes that students should be able to demonstrate as a result of the instruction; (3) brainstorm characteristics of each attribute; (4) write narrative descriptions for the levels of performance for each attribute (p. 32).

9. Rubric for this study

10. Using the Rubric 3 raters Achieving kappa
Two professional archivists with at least five years of experience teaching undergraduates and me
Important for increasing the reliability of the scores
Achieving kappa
Final results
Our Fleiss Kappa for the pretest was and .788 for the post-test, both of which are considered excellent strengths of agreement.

11. Statistical Tests Non-parametric tests
Useful for analyzing ordinal and nominal data
Great for small samples
Tests used:
Mann Whitney U Test for Independent Samples
Results
The tests confirmed that the null hypothesis could be rejected with 95% confidence
Students in the treatment group did better on the post-test after participating in archival instruction. In fact, there is a statistically significant difference between the two groups’ scores for each of the criteria.

14. Discussion The Research Skills Problem Disappointing results
Possible factors
Students not actively engaged during instruction
Follow-up assignment involved pre-selected documents – no chance to practice searching for and identifying documents
Literature on undergraduate information-seeking behavior
The post-test scores of the treatment group point to an improvement in all of the skill categories except Research Skills as a result of the archival instruction those students received. In fact, both groups of students performed about the same in this skill category with not one student in either group achieving an exemplary score. This finding can possibly be attributed a couple factors. Firstly, although a component of their archival instruction focused on searching for archival collections, the students absorbed that information passively. Few, if any, took notes during the brief presentation and none were actually engaged in searching for primary sources. Secondly, students were taught how to search for collections in local library and archives catalogs. Not surprisingly, the majority of the students’ responses focused on finding sources in the local repository. Thirdly, the follow-up assignment the students received was based on analyzing pre-selected electronic resources and did not require the students to return to the archives to conduct research. This did not give them an opportunity to reinforce the searching skills presented during archival instruction.
The fact that the students in this study did not possess sophisticated research skills even after archival instruction is supported by the literature on undergraduate information-seeking skills. Many researchers have pointed out that students exhibit elementary searching and organization skills (Leckie 1996, Maughan 2001, Quarton 2003). A recent study of the “Google Generation” commissioned by the British Library and JISC (Joint Information Systems Committee) (Rowlands 2008) synthesized decades of literature about young people’s information seeking behavior and concluded that today’s undergraduates are not as “web-literate” as conventional wisdom assumes. Their unquestioning dependence on the accuracy of search engines like Google and Yahoo is consistent with research on how students judge the credibility of online resources (Hung 2004, Rieh & Hilligoss 2007, Hilligoss & Rieh 2008). The study also emphasized that teaching information literacy skills during the formative school years is more effective than introducing these concepts to college students who have already formed their online information seeking behavior preferences.
In order for students to learn archival research skills, they not only need to be exposed to the process of using an archives, but they also need to practice searching for and identifying materials. This is where the archival repository becomes a learning laboratory for students in the humanities and social sciences. Archivists are educators in more than one sense. They provide instruction about primary sources and their collections to students and other researchers. They also allow researchers to explore and provide guidance throughout the process. Students really need this if they are going to fully grasp the concepts archivists want to teach them, including the differences between primary and secondary sources as well as how to find and critically interpret them. Students in history courses such as the one described in this study, learn to interpret primary sources through pre-selected, published materials. They may be able to identify the characteristics and elements of those documents and place them in a larger historical context. However, as this study has demonstrated, students have difficulty understanding how to identify and locate sources in order to answer a research question. They will not learn these valuable research skills without ample time and opportunity to practice them in the archives.

15. Archival Literacy Scale
Elements
4 categories of the rubric:
Observation
Interpretation
Evaluation
Research Skills
Testing for reliability
Results
The four criteria or categories of analysis in the rubric designed for this study encompass a set of skills associated with archival literacy. These skills include the ability to identify important elements of primary sources and to place them in an historical context. They also include an ability to make inferences about primary sources, validate them, and identify how to search for and retrieve related sources. In theory and in practice, these are related skills. I performed a test of statistical reliability in order to examine the underlying structure among these four skills.
In this case, reliability refers to a measure of the internal consistency among the items (or indicators) and can be calculated through the Cronbach’s alpha coefficient (De Vaus 2002). This alpha ranges between 0 and 1 and the higher the alpha, the more reliable the scale. In practice, an alpha of at least 0.7 is accepted as reliable (De Vaus, p. 184).
I hypothesized that these skills would be highly correlated because generally, students received similar scores on each of the categories. In other words, if a student had fair observation skills, he or she would most likely have fair critical thinking and research skills. This was definitely the case on the pretest document analysis exercise. The overall Cronbach’s alpha was .792 for all the categories, which is highly reliable (see Table 4.8). This high alpha score indicates that the categories are indistinguishable on the pretest, leading to the conclusion that there is a reliable concept made up of these four skills I am calling “archival literacy skills.”

16. Conclusions Limitations to the generalizability of results –
Need to replicate the study to increase the reliability of the instruments
Wish for better instructional design for instruction component
Need for more communication and collaboration between archivists to develop and share curricula and standardized tools to assist in instruction and assessment

17. Thank You
For more information about the study described in this presentation, please contact:
Magia Krause