1. Creating a Comprehensive Bank of Assessments for Use in Applied Statistical Methods (Stat 1000)
Brian Leventhal, Psychology in Education PhD Expected 2017
Dr. Nancy Pfenning, senior Lecturer, Department of Statistics

2. Brian Leventhal (brl38@pitt.edu); Nancy Pfenning (nancyp@pitt.edu)
Introduction
Propose a transformation of assessment materials in Stat 1000 in order to consolidate and enhance existing materials
Goal: ensure fair and appropriate content for all instructors – focus on new hires
Begin: examination of questions of existing assessment materials developed by various faculty to date.
Use assignments, projects that require statistical software, quizzes, midterm and final examinations
Goal: organize a coherent collection of materials that make it possible for a less experienced lecturer to “hit the ground running” with high-quality assessment tools that are consistent with Stat department’s goals for instruction in that course
Brian Leventhal Nancy Pfenning

3. Brian Leventhal (brl38@pitt.edu); Nancy Pfenning (nancyp@pitt.edu)
Background
Introductory statistics classes service a large number of students from various backgrounds and different fields.
Stat 200, Stat 1000, Stat 1100
Increased reliance on adjunct instructors
Spring 2016: only 3 out of 15 lectures taught by full-time statistics faculty
Stat 200 – Dr. Nancy Pfenning has successfully transformed and consolidated assessment materials through use of her text and accompanying slides, etc.
Next Step: Stat 1000
Fall – 5 lecture sections offered; Spring – 4 lecture sections; Summer – 2 lecture sections
Future: Stat 1100
Brian Leventhal Nancy Pfenning

4. Teacher Evaluations – Spring 2016
Adjunct 1: OMET
OMET: “The instructor evaluated my work fairly”: mean 2.62 out of 5
Overall teaching effectiveness: Mean 1.90
Adjunct 2: Mid-semester survey
“Because the lecturer is new to the university it seems as though the university has not prepared him enough…”
Brian Leventhal Nancy Pfenning

5. Teacher Evaluations Spring 2016 cont’d
Adjunct 3: Mid-semester survey
It seems like he was tossed to the wolves at the last minute and given some class materials and told to go run this class… We have had sporadic quizzes(2 so far) and they have been on material that is several lectures back. We could be on lecture 12 in lecture, talking about lecture 7 in recitation and doing homework on lectures 4 and 5 and a quiz on lectures That seems how this class has been going. It was like that one week for recitation/homework/quiz. Im honestly thinking about sitting down with someone in the STAT department to talk about this class. He's well prepared for class in a sense of knowing the material and teaching it. He is not well prepared for class in regards to the other areas like quizzes, homework, recitation, stat lab, etc
Adjunct 4:
OMET: “The instructor evaluated my work fairly”: mean 4.73 out of 5
Overall teaching effectiveness: Mean 4.18
Brian Leventhal Nancy Pfenning

6. Brian Leventhal (brl38@pitt.edu); Nancy Pfenning (nancyp@pitt.edu)
Step #1
Survey of adjunct lecturers about usefulness of materials provided
Provided to three adjunct Spring 2016 Stat 1000 lecturers as well as two adjunct Stat 200 lecturers for comparison
Assess:
Usefulness of materials provided
How easy or difficult it was to adapt various materials
Will be administered at end of Fall 2016 and End of Spring 2017 as well
Brian Leventhal Nancy Pfenning

7. Brian Leventhal (brl38@pitt.edu); Nancy Pfenning (nancyp@pitt.edu)
Step #2
Material has been gathered from experienced lecturers: Dr. Carl Bodenschatz and Dr. Kehui Chen.
Midterm examinations
Final examinations
Quizzes
Homework assignments
Exam practice material
Lab assessments
Course syllabi: frequency of quizzes, examinations, lab assessments, homework assignments
Student grades will be collected in order to make a comparison of final grade distributions for students from various lectures of STAT 1000.
Brian Leventhal Nancy Pfenning

8. Brian Leventhal (brl38@pitt.edu); Nancy Pfenning (nancyp@pitt.edu)
Step #3
Ongoing Discussion with experience lecturers: Dr. Carl Bodenschatz and Dr. Kehui Chen
Discussions will include gaining recommendations for
Homework assignments
Laboratory assessments
Number of examinations
Other useful ideas relevant to incoming adjuncts or new full-time faculty
What material is considered most important for assessment in final exams
Brian Leventhal Nancy Pfenning

9. Brian Leventhal (brl38@pitt.edu); Nancy Pfenning (nancyp@pitt.edu)
Step #4
Handing off some prepared materials to Fall 2016 Stat 1000 adjuncts
Dr. Carl Bodenschatz will be teaching two lectures
Brian Leventhal will most likely be teaching one lecture
A new adjunct will probably be teaching the final section of STAT 1000.
Brian Leventhal Nancy Pfenning

10. Brian Leventhal (brl38@pitt.edu); Nancy Pfenning (nancyp@pitt.edu)
Step #5
Ongoing review of Fall 2016 Stat 1000 materials from Dr. Carl Bodenschatz
Engaging in thorough discussions about material on current assignments and assessments to see
Which items work best
What item styles (eg. multiple choice, short-answer, etc.) are suggested
Other aspects Dr. Bodenschatz thinks are essential for organization of the course
Brian Leventhal Nancy Pfenning

11. Brian Leventhal (brl38@pitt.edu); Nancy Pfenning (nancyp@pitt.edu)
Step #6
Creation of annotated test bank for final exams
Will contain a variety of constructed response and selected response items
Items will range across the taxonomy of educational objectives
Consist of statistical basics, choosing the correct analysis procedure, and interpreting output from statistical software
Items will be classified by topic and type
After classification of items, topics will be assigned a recommended percentage of final exam coverage, so major topics are given roughly the same weight on various instructors’ final exams
Brian Leventhal Nancy Pfenning

12. Brian Leventhal (brl38@pitt.edu); Nancy Pfenning (nancyp@pitt.edu)
Step #7
Handing off more comprehensive packet of materials to Spring 2017 Stat 1000 adjuncts
New instructors expected to teach lectures of Stat 1000 will be provided with a list of
Final examination assessment items
Sample syllabi
Sample homework assignments
Sample lab assignments
Sample midterm examinations
Course organization recommendations.
Brian Leventhal Nancy Pfenning

13. Brian Leventhal (brl38@pitt.edu); Nancy Pfenning (nancyp@pitt.edu)
Overall Progress
Throughout the pilot study we intend to monitor how much, and, which type of material the instructors use. 
We intend to discuss this with the instructors to understand their thought process so that we can improve and cater to the needs of new instructors
OMET course evaluation and student grade distributions will be compared
Help inform us which material is most beneficial to the adjuncts. 
This information will be used to make recommendations in our user guide we will provide to the adjuncts.
. [Dr. Pfenning will screen the OMETs herself and provide Brian with information that does not breach confidentiality of info about instructors’ performances]
Brian Leventhal Nancy Pfenning

14. Objectives and Assessments #1
Easier and more straightforward preparation for new lecturers of Stat 1000
Assessed through survey (Step #1)
Example items will include:
The material provided made the development of my STAT 1000 course easier.
(strongly disagree, disagree, neutral, agree, strongly agree)
The percentage recommendations for material on the final exam influenced the time I spent on material in lecture.
The material provided included a wide enough range of items to select from.
Working with the provided material, I was still able to develop a course that suited my teaching style.
Example items will include:
The material provided made the development of my STAT 1000 course easier.
(strongly disagree, disagree, neutral, agree, strongly agree)
Justification: One goal of this study is to make the development of the STAT 1000 less overwhelming and easier for the incoming adjuncts. As some adjuncts are hired on a part-time basis, they may not have the resources to put together a well-developed course. This package is intended to ease the process and reduce the time it takes to develop a well-rounded course.
The percentage recommendations for material on the final exam influenced the time I spent on material in lecture.
Justification: Using recommendations from experienced lecturers for which material is most important, new lecturers should spend more time focusing on these topics. Without explicitly stating how much time must be spent on each and every section of the textbook, the hope is that the recommended percentage will let adjunct instructors know what is deemed most important for the department.
The material provided included a wide enough range of items to select from.
Justification: The point of the package is to provide recommendations for items and to ease instructors’ development of their lectures. However, we wish to also give flexibility to the instructors and do not want them to be too constrained by our recommendations.
Working with the provided material, I was still able to develop a course that suited my teaching style.
Justification: We understand that every instructor has his or her own teaching style. In our view, it is best not to hinder this style to a point where the instructors feel constrained and are limited. Because each instructor has strengths and weakness, we hope to enable them as individuals to make the most of their strengths.
Brian Leventhal Nancy Pfenning

15. Objectives and Assessments #2
Better quality of materials used in Stat 1000, especially by less-experienced adjuncts
This will mainly be assessed through mid-semester lecturer survey and end-of-semester OMETs.
Example adjunct:
Scored 2.05 out of 5 on presenting course in organized manner
Scored 2.62 out of 5 on evaluating students work fairly
One of our objectives is to ensure that such low levels do not occur again in future Stat lectures.
Although these surveys don’t ask students specifically about their instructors’ resources, questions about the instructor’s level of organization and about being evaluated fairly do provide us with relevant feedback.
Brian Leventhal Nancy Pfenning

16. Objectives and Assessments #3
More uniform instruction in all Stat 1000 lectures
Checking for less disparity compared with pre-transformation scores
Scores on final exams will also be analyzed
Brian Leventhal Nancy Pfenning

17. Challenges and Responding Plans
We recognize that individual instructors have their particular strengths and weaknesses.
Even if the prepared materials are considered optimal, a lecturer with poor organizational skills might not be able to incorporate them effectively.
Especially during the first two semesters of this project, efforts will be made to develop a user’s guide that anticipates and mitigates as many teaching pitfalls as possible.
Brian Leventhal Nancy Pfenning

18. Sustainability and Scalability
All the materials developed as part of this transformation will be catalogued and stored (using a Box folder) in such a way that they will be as useful and accessible as possible in future semesters.
Access to the Box will be limited to Stat 1000 lecturers. This transformation can also serve as a model to streamline a similar endeavor in the future for our Stat 1100 course.
Brian Leventhal Nancy Pfenning

19. Brian Leventhal (brl38@pitt.edu); Nancy Pfenning (nancyp@pitt.edu)
Conclusion
Students and instructors alike will benefit from the implementation of this project made possible by dB-SERC. We look forward to reporting progress during the coming fall and spring semesters.
Questions? Tips on Project? Please contact:
Brian Leventhal –
Dr. Nancy Pfenning –
Brian Leventhal Nancy Pfenning