University Park IST/SRA 440W Peer Evaluation of Teammates Spring 2017 Qualtrics Survey Results from University Park IST/SRA 440W Students (n = 121) March 27th 2017, 8:26 pm EDT

Q11 - What is your class standing? # Answer % Count 1 First-year student 0.00% 2 Sophomore 3 Junior 4.13% 5 4 Senior 95.87% 116 Total 100% 121

Q1 - Consider your College of IST courses that include team assignments (e.g., projects, papers, presentations, etc.). With what frequency have you been asked to evaluate team members?

Q1 - Consider your College of IST courses that include team assignments (e.g., projects, papers, presentations, etc.). With what frequency have you been asked to evaluate team members? # Answer % Count 1 > 75% of my courses that include teamwork require me to evaluate team members 73.55% 89 2 50-75% of my courses that include teamwork require me to evaluate team members 24.79% 30 3 < 50% (and at least in one) of my courses that include teamwork require me to evaluate team members 1.65% 4 0% of my courses that include teamwork require me to evaluate team members 0.00% Total 100% 121

Q2 - Check all the tools that you have used for the purpose of evaluating team members:

Q2 - Check all the tools that you have used for the purpose of evaluating team members: # Answer % Count 1 CATME (Comprehensive Assessment for Team-Member Effectiveness) 76.86% 93 2 World Campus Peer Evaluation tool 14.05% 17 3 Online or paper evaluation created by the instructional team (i.e., the instructor and/or TA and/or LA) 93.39% 113 4 Other 12.40% 15 Total 100% 121

Q3 - What other tool(s) have you used to evaluate team members? (6) Just write names down and describe if they did well—paragraph for each person (even in Word) (4) Any online survey tool (e.g., Google survey) (3) Angel, Canvas surveys (1) Custom built quizzes

Q6 - Do you think it is appropriate for students to evaluate their team members when working on team assignments?

Q6 - Do you think it is appropriate for students to evaluate their team members when working on team assignments? # Answer % Count 1 Definitely yes 57.02% 69 2 Probably yes 29.75% 36 3 Might or might not 10.74% 13 4 Probably not 2.48% 5 Definitely not 0.00% Total 100% 121

Q7 - In your experience in College of IST courses, what is the most common weight that team member peer evaluations are given toward the team assignment grade?

Q7 - In your experience in College of IST courses, what is the most common weight that team member peer evaluations are given toward the team assignment grade? # Answer % Count 1 > 50% of the team assignment grade is determined by team member peer evaluations 5.83% 7 2 Between 26 and 50% of the team assignment grade is determined by team member peer evaluations 12.50% 15 3 Between 11 and 25% of the team assignment grade is determined by team member peer evaluations 25.83% 31 4 Between 1 and 10% of the team assignment grade is determined by team member peer evaluations 36.67% 44 5 Although we are asked to evaluate our peers, the evaluation does not count toward the team assignment grade; it is counted as a separate grade altogether 18.33% 22 6 Although we are asked to evaluate our peers, the evaluation does not count toward any part of our grade 0.83% Total 100% 120

Q8 - Considering tools that you have used to evaluate your team members, which features are important to you? (Check all that apply.)

Q8 - Considering tools that you have used to evaluate your team members, which features are important to you? (Check all that apply.) # Answer % Count 1 Anonymity 83.47% 101 2 Rating scale that includes a broad range of points (i.e., at least 6 or 7 points, so that a 1-point deduction does not automatically reduce a student's grade from an A to a B) 74.38% 90 3 Option to rate myself as well as my team members 38.02% 46 4 Comparison of my self-rating with my team members' ratings of me 24.79% 30 5 Average scores from my team members 33.06% 40 6 Option to add written comments 65.29% 79 7 Graphics that make the data easy to understand 11.57% 14 8 Option to evaluate my team as a whole unit 39.67% 48 9 Ease of using the tool, in general 60.33% 73 10 Other 0.00% Total 100% 121

Q12 - What is the name of the tool that you have used most often to evaluate team members? # Answer % Count 1 CATME (Comprehensive Assessment for Team-Member Effectiveness) 28.10% 34 2 World Campus Peer Evaluation tool 0.83% 3 Online or paper evaluation created by the instructional team (i.e., the instructor and/or TA and/or LA) 70.25% 85 4 Other Total 100% 121

Q13 - Briefly, what do you like about CATME; and what do you dislike about CATME? (15) Very easy to use (4) It works well (2) Anonymity It allows me to leave written comments I like that Catme can send email reminders to complete the evaluation. CATME is a good tool, but it tends to lump a large amount of tasks into one survey. "Team member was a leader, exhibited outstanding organization, and was punctual." Things like this are okay, but maybe a team member was a great asset to the team, but was late on occasion. It provides a leaflet view so I can review all member quickly which is a plus. There is not enough color difference between each leaflet though, so I may lose my place at times I wish more professors used it to its full ability. You can align students with what times work best for them and which characteristics will work the best together.

Q13 - Briefly, what do you like about CATME; and what do you dislike about CATME? Briefly, what do you dislike about CATME? (4) The user interface. It seems dated (2) It should give you the ability to specific comments for each team member I don't like the fact that the same general format is used for a large variety of classes that are very different and have different types of teams. Sometimes hard to tell what the highest rating actually is - the in between scores are confusing does not allow you to rate your group evaluation as a whole it's really "wordy" some of the questions seem very repetitive I think it's mostly useless, because the metrics are usually artificial. If I have a problem or a compliment with a team member it is going to be something specific, not a vague comment about them. I dislike CATME uses a simple system to evaluate team members. There is not a lot of in-depth analysis of other team members. The evaluations are a little unfair on CATME. This is probably more of a personal issue, but I didn't like that the only way we could access it was through a link in web mail. It would be nice if these types of surveys could be an assignment built into a course management system.

Q14 - Briefly, what do you like about the evaluation tool created by the instructional team? Briefly, what do you like about the tool created by the instructional team? (20) Straight-forward (simplicity) and easy to use (and fast) (14) Can use my own words (comments) to evaluate each team member (14) Instructor can tailor to our specific class needs (8) I like the rating system (e.g., On a scale of 1-10) rating—offers more options to rate students (5) Anonymity (1) Instructor uses mid-semester evaluations and end of semester evaluations (1) Easy to find in Canvas or ANGEL (1) it works well

Q15 - Briefly, what do you dislike about the evaluation tool created by the instructional team? (8) Hassle: of downloading and uploading documents for multiple team members (6), of looking up team members to complete the forms (1), or of logging into another system that the instructor decided to use to create the form (1) (7) Too much writing is required/takes too much time to complete (5) Not enough options to rate team members (e.g., want 1-7 or 1-10); and not well-defined scales (4) We don’t see the feedback from our team members (3) Form is too generic and general (2) Don’t know how the scores will be used (2) Sometimes, questions are redundant (1) Lack of anonymity

What did World Campus students think?

What did World Campus students think?

Q10 - If there is anything else you would like to share about your experiences with evaluating your peers on team assignments, please add your comments here. If there is anything else you would like to share about your experiences wi... I wish more of my classes did some kind of evaluation within IST and within the university overall. I also wish we got to see the feedback on the peer review...I don't think I've ever seen the feedback I've received from my team members. That completely eliminates the true purpose of the feedback if I never get to see it. Comments should be required in ratings to ensure there is no confusion about scores. Short peer review with easy to use rating system for self and team members is nice Honestly peer evaluations are really not worth it unless you have someone that really stinks and hasn't done anything. I think I have only not given someone full credit once in four years, and my experiences and team members have all been on a spectrum. Sometimes I don't feel like the ratings really accomplish anything given that in past teams, no effective action has been taken against poorly performing team members, and due to the grading structure of IST courses, individual participation generally accounts for very little of the grade, creating the conundrum that affects IST at large: many students coast by on the work of others for their 4 or so years in the college, graduating with unearned yet passing grades. This is something that all IST majors I know have complained about (not an exaggeration). I did not like using catme. I used it for one class but I felt the tool was unnecessary. I would recommend directly integrating a tool into Canvas so people do not need to redirect outside of Canvas to fill out evaluations.

Q10 - If there is anything else you would like to share about your experiences with evaluating your peers on team assignments, please add your comments here. If there is anything else you would like to share about your experiences wi... I think having team evaluations impact grades encourages people to not be honest with their assessments. Part of me is against it, because students don't have a stake like a professor does when giving a "fair" grade. However part of me feels like gaining the respect of your peers is all together another professional trait that you need to learn to work on no matter what field you're in. Getting positive peer reviews is one way for you to assess this. Sometimes professors are made when you give everyone the max points or whatever, even though it is true. One time a professor made us write what each team member did but it was difficult to remember everything, and a lot of the time people were working on the same stuff together. If there are multiple peer evaluations sometimes students who receive a bad peer review from their peers on the first review will give their other teammates lower scores because they were receiving lower scores to get back at their teammates. I think it is really hard to evaluate teammates especially if it is something that the work is hard to determine It seems like people do not really take it seriously I think that if the weight of the grade was less, it would be easier to rate someone truly. If the grade is weighted a lot I usually will give them a decent rate to not ruin their grade

Q10 - If there is anything else you would like to share about your experiences with evaluating your peers on team assignments, please add your comments here. If there is anything else you would like to share about your experiences wi... most students give team members the higher ratings since they know it affects the other student's final grade. students feel bad hurting other's grades even if they deserve a low score. I do not think students should rate each other. If there is a significant enough problem a team member, that should be brought to the professor's attention separately. As a veteran, and an older student I was always almost forced into the leadership slot which made things hard from SRA 111- SRA 440. I felt like I was herding cats all semester and my labs/HW/study time etc. suffered because I put in so much extra effort as the team leader. There needs to be either no team leader or rotating team leaders due to so many students knowing that being the team leader doesn't help your own grade at all, but actually makes it suffer. Why do extra work trying to get some 19 year old kid off of "League of Legends" and do his part when I could just be doing the equivalent. I know with some evaluations people just give everyone good ratings because they don't want to rat them out.

Q10 - If there is anything else you would like to share about your experiences with evaluating your peers on team assignments, please add your comments here. If there is anything else you would like to share about your experiences wi... The issue is peer evaluations feel like snitching. Its hard to rate someone below a 50% mark and there's an expectation to rate everyone kindly, even in the face of low performance. Most people don't recognize that they did not contribute. They believe their teammates just weren't fair. I think there's also a bias toward "good" reviews because the belief is the instructor won't respond tactfully. I appreciate these as they provide an excellent opportunity to reflect on the work done by fellow teammates. Knowing that this type of evaluation is part of the project helps motivate many members who who would be less motivated otherwise. Peer reviews can be conducted easily with a few questions. The main question should be "do you feel that (person x) should receive the same grade as the rest of your team members?" It would be nice to receive feedback about personal performance from other students in anonymous form rather than not getting any potential constructive criticism. I am not a big fan of these analysis tools, as they often force students to impact each other's final grade. This often leads to reviewers not being honest about how effective team members are, because they don't want backlash from group members, and also do not want to be responsible for failing another student's participation. This makes group projects problematic, because there will always be slackers, and they will generally not get caught or change their behavior because of other group member's kind nature.

Q10 - If there is anything else you would like to share about your experiences with evaluating your peers on team assignments, please add your comments here. If there is anything else you would like to share about your experiences wi... If the person is really trying to give an honest peer review, I think that a basic scale and a writing box are enough. None - I think it's good to evaluate team members. It's also hard to keep everyone happy in doing so. Anonymity is important because some students are afraid to write truthful evaluations about their peers. I think every class should do it. Peer evaluation is not very effective because the member who did not contribute much still gets away with doing absolutely nothing for the team. I think that all peer evaluations should be done at the end of the project. While mid-semester peer evaluations are good tools to check up on teammates, if a teammate gets a bad score, it may hurt their motivation to work. This would mean the tool hurts more than it helps. Sometimes I would like to just be able to write a paragraph about each team member touching upon different attributes about them like attendance, team work, etc... Answering rating questions can be ambiguous at times and not accurately portray how I feel about certain members,