Assessment of Teaching and Learning with Technology at Seton Hall University Janet Easterling, Institutional Research Danielle Mirliss, Senior Instructional Designer Shayle Abelkop, Instructional Designer Copyright Abelkop, S., Easterling, E., and Mirliss, D. (2007). This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.

Why Mobile Computing “Seton Hall University’s Mobile Computing Program is designed to break down the barriers of space and time and allow all our students to fully participate in our scholarly community and achieve high levels of performance.” Mobile Computing Mission Statement This has been our main impetus for assessment – we have been at this for some time – program in place since 1996

Mobile Computing Timeline 1994 – 1996: Development of SHU Strategic Plan 1995 – 1997: Mobile Computing Pilot Project with small cohorts of students in Business School 1997: Mobile Computing Pilot Project expands to 300 incoming Freshman 1998: Mobile Computing Program launched for all incoming Freshman 2001: Mobile Computing Program expanded to select graduate programs; campus wireless network implemented 2002: SHU graduates first mobile class 2004: Tablet PC Pilot Project Poll the audience – who is responsible for doing assessment on their campus? Who is actually engaged in assessment of technology? Have audience describe where they are and their interest in this topic?

Mobile Computing @ SHU Innovative academic program involving three components: Access: The University licenses student use of a laptop computer as part of tuition and fees. Curricular Integration: The University encourages and supports faculty use of technology in teaching and learning. Network and Support Services: The University provides the infrastructure that enables effective use of technology in teaching and learning. This is how we have viewed our mobile computing program – the inspiration for our assessment efforts. From the T&L perspective the curricular integration is very important.

Mobile Computing Assessment Began in 1996 Student Survey Historically focused on: Overall Satisfaction Use of the Technology Impact on Effective Teaching Practice 2005-2006 Renamed Teaching and Learning with Technology Survey Emphasis on Student Use and Effective Integration of Technology Make sure to mention survey fatigue – mixing the technical with the pedagogy . Talk about importance of team - stakeholders

Early Efforts/Challenges In an effort to assess everything the survey grew to monstrous proportions Felt a need to document clearly positive outcomes of efforts to infuse Then compelled to monitor positives Added questions but also kept all the questions… Slide on what to assess – cannot ask everything

Better but not there yet… Survey questions were narrowed to emphasize teaching and learning Exploratory approach to achieve better questions– open-ended/focus groups Smaller number, better questions Application of results to inform practice Slide on what to assess – cannot ask everything

Where we want to be… Ask questions that are meaningful to all stakeholders (even students!!) Craft a survey each year based on needs in a changing context Disseminate information to campus and beyond Include additional stakeholders to drive questions Other people are driving what we want to look at

Results from along the way… Quantitative – student responses on usage, gains, preferences & benefits Qualitative – opinions, details of perceived benefits and barriers Other people are driving what we want to look at

T & Learning with Technology Students Quantitative highlights Hours each week on-line Q3 Usage of IT to .. Q4 IT preferences in courses Q5 IT improves collaboration, feedback, control? Q6 IT improves my learning Q7 IT feature most valuable Q8 Gains d.t. on-line Bb/web Q12 Opps 7prins qual learning Q13 Janet

Learning with Technology Student Survey administered April 2006 to a random sample of freshmen and juniors oversampling for males and for students identifying as non-white 253 responses, overall response rate 30% Higher rates for Women, Residents, FR Lower rates for Men, Non-White, Commuting Students, JR

1 –. Hours each week on-line Q3 How many hours per week would you estimate you spend… Most frequent response 6-10 hours / week on-line (www/e-mail/..) for academic work avg 11 hours ~1 in 2 students (a little under) specify more than 10 hours a week ~1 in 5 students (a little over) 5 or fewer hours a week ~1 in 10 students (a little over) 21 or more hours a week 45% more than 10 hours (11% 21+) 23% 5 hours or less ( 4% zero)

1 –. Hours each week on-line Q3 How many hours per week would you estimate you spend …. Most frequent response 6-10 hours / week on-line (www/e-mail/..) for non-academic work avg 13 hours ~1 in 2 students (a little over) specify more than 10 hours a week ~1 in 4 students 5 or fewer hours a week ~1 in 4 students (a little over) 21 or more hours a week 55% more than 10 hours (27% 21+ hours) 25% 5 hours or less (11% zero)

~3 in 5 students specify 21 or more ! ~1 in 4 students 10 or fewer 1 –. Hours each week on-line Q3 How many hours per week would you estimate you spend.. Summed responses for hours / week on-line (www/e-mail/..) in total avg 23 hours ~3 in 5 students specify 21 or more ! ~1 in 4 students 10 or fewer ~1 in 10 students 41 or more !! 60% more than 20 hours (12% 41+ hours) 25% 10 hours or less ( 2% 5 or less)

2 – Use of IT for Academic work Q4 On average how often do you use Information Technology to.. Daily/Weekly/Monthly/Rarely/Never Frequency of use of IT for academic work to complete course assignments 77% D/W e-resources from Library web 48% D/W e-resources from other sites 72% D/W (other than a library web site)

2 – Use of IT for Academic work Q4 On average how often do you use Information Technology to.. Daily/Weekly/Monthly/Rarely/Never Freq of use of IT for academic work to work in teams during class 62% D/W/M outside of class 70% D/W/M (to complete course assignments)

3 – Student prefs for use of tech in courses Q5 I prefer taking courses that … first choice, courses that use a moderate level of technology 42% (email, several ppt presentations, some on-line activities/content) however, almost tied ! use technology extensively 39% (class lecture notes on-line, ppt, computer simulations, streaming video) third choice use limited technology 13% (professor email, limited ppt use)

4 – Impact of IT Q6 Compared to courses that use little or not technology, the use of technology … (agreement) use of IT allows me to Take greater control of my course activities planning, apportioning time, awareness of success/failure 89% (42% strongly agree, 37% agree) use of IT results in More prompt feedback from my instructor 85% (40% strongly agree, 45% agree) use of IT helps me Better communicate and collaborate with my classmates 80% (38% strongly agree, 42% agree)

has Improved my Learning 80% (strongly agree or agree) 5 –IT Improves Learning? Q7 The use of technology in my courses has improved my learning… (agreement) use of IT in my Courses has Improved my Learning 80% (strongly agree or agree) ------------------------------------------------------------------ 21% strongly agree 59% agree 15% neutral 3% disagree 2% strongly disagree

benefit from using technology 68% convenience / saves time 6 – IT Most valuable benefit Q8 Which of the following benefits from using technology in your classes are most valuable to you… single most valuable benefit from using technology 68% convenience / saves time 19% helps manage class activities 9% improves my learning 3% other 2% none

Course selected by the student 7 – Perceived gains from on-line (Bb/web) use Q12 As you answer the following questions, think about one particular course from among those you are taking this semester that, in your opinion, makes effective use of technology to support teaching and learning… In this course, what was the effect of having the following on-line features (available through Blackboard or your professor’s web site)? Course selected by the student as one that makes effective use .. (student reasons for selecting – Danielle Mirliss will present these)

Course selected by the student 7 – Perceived gains from on-line (Bb/web) use Q12 As you answer the following questions, think about one particular course from among those you are taking this semester that, in your opinion, makes effective use of technology to support teaching and learning… In this course, what was the effect of having the following on-line features (available through Blackboard or your professor’s web site)? Course selected by the student as one that makes effective use .. 59% Mostly Undergraduate – FR/SO 39% Mostly Undergraduate – JR/SR

In course selected by the student 7 – Perceived gains from on-line (Bb/web) use Q12 As you answer the following questions, think about one particular course from among those you are taking this semester that, in your opinion, makes effective use of technology to support teaching and learning… In this course, what was the effect of having the following on-line features (available through Blackboard or your professor’s web site)? In course selected by the student as one that makes effective use .. Improved Learning %responses Online readings & links to materials 67% Online discussion board 47% Sharing materials with students 45% Improved Management of Class Activities Online syllabus 85% Ability to turn in assignments on-line 73%

In course selected by the student 7 – Perceived gains from on-line (Bb/web) use Q12 As you answer the following questions, think about one particular course from among those you are taking this semester that, in your opinion, makes effective use of technology to support teaching and learning… In this course, what was the effect of having the following on-line features (available through Blackboard or your professor’s web site)? In course selected by the student as one that makes effective use .. Positive Experience 0/1 Mean Online syllabus .89 Online readings & links .86 Online turn in assignments .79 Online sharing materials .70 Online get grades, comms .66 Online track your grades .64 Online discussion board .63

In course selected by the student 8 – Opportunity for 7-principles learning env’s Q13 As you answer the following questions, think about one particular course from among those you are taking this semester that, in your opinion, makes effective use of technology to support teaching and learning… In this course, how often were you able to do or experience the following)? In course selected by the student how often able to do or experience .. Student Faculty Contact %often {very} Have contact with professors 80% on course content Diverse ways of learning % often {very} Explore course materials .. 75% Demonstrate what I have learned .. 63% .. in different ways

In course selected by the student 8 – Opportunity for 7-principles learning env’s Q13 As you answer the following questions, think about one particular course from among those you are taking this semester that, in your opinion, makes effective use of technology to support teaching and learning… In this course, how often were you able to do or experience the following)? In course selected by the student how often able to do or experience .. Cooperation among students %often {very} Spend time with other students on .. 55% Feedback Get prompt feedback on performance 73% Observe & record my own progress 57% Time on Task Edit and improve my projects / assign. 60%

8 – Opportunity for 7-principles learning env’s Q13 As you answer the following questions, think about one particular course from among those you are taking this semester that, in your opinion, makes effective use of technology to support teaching and learning… In this course, how often were you able to do or experience the following)? Does positive experience with related on-line activity relate to frequency of opportunity for high quality learning environments? Cooperation among students %often {very} Q13b Spend time with other students on .. 55% Positive experience with on-line Q12 Online readings & links .86 .23 Online sharing materials .70 .32 Online discussion board .63 .14 mean correlation with 13b

9 – Strengths, Challenges Quantitative data More gen’l observations .. come back to.. Strengths .. results s-f contact .. (as in prior years) s-s interaction .. (newly revealed) Challenges .. data ? serious responses .. (given incentive) ? generalizable .. (given response %)

10 –Teaching & Learning with Technology survey assessment continued Next .. Reviews of qualitative - Danielle & Shayle helps .. hinders .. effective use .. +Focus Groups to probe, explore - “effective uses” of technology – examps .. - gains from interactions – which online resources.. Survey Qs next cycle that probe effective use in terms meaningful to student

Qualitative Responses

Technology helps learning by… Providing anytime, anywhere access Organizing information Facilitating communication among students/professors Broadening research/resource materials Efficient methods of submitting assignments Making access equitable Having projectors in class Exercising technical skills

What students said… “Giving us access to certain resources that benefit our learning experience as a whole” “Giving students an outlet to learn valuable skills” “Making it easier to contact professors and fellow students for help, notes, etc.” “Technology is always available and accessible when I need it, whether I want to take notes in class with my laptop or use resources available through the school’s homepage” “With all the information that is now provided via the internet, the technology we have guides us to easy access for attaining this information.”

Technology prevents learning by… Providing distractions Technical issues Too much reliance Facilitating plagiarism Less personal contact w/ students “At Seton Hall, technology can prevent learning because during lectures some students may be too busy using instant messenger instead of paying attention to the professor.”

What students said… “Being a distraction for those who do not know when to separate from it at needed times.” “Having access to cheating methods.” “Technology can prevent students from learning because they are able to surf the net while in class as well as talk on AIM.” “Many students bring their laptops to classes and become distracted by them because it gives them something to do besides paying attention.”

Why did you choose this course as one that makes effective use of technology? Assignments – ability to complete and submit assignments electronically “This course not only provides me with a syllabus online but with other assignments that are to be done online.” Collaboration – use of technology to communicate and co-author assignments “I thought it allowed me to work better with my classmates.”

Why did you choose this course as one that makes effective use of technology? Convenience – enables students to keep track of work and updated materials “Kept me up to date with my course grades and assignments.” Feedback – provides a way for faculty to give feedback on assignments “Many of our assignments are turned in online via our discussion board and we are given feedback on certain assignments from our professor as well.”

Why did you choose this course as one that makes effective use of technology? Frequency – course uses technology more than other courses “Because technology is used for every aspect of the class: assignments, quizzes, lectures, discussion board, etc.” Information Literacy – ability to access resources from library and Internet “We use the library database every day.”

Why did you choose this course as one that makes effective use of technology? Learning – contributed to students’ understanding of course material “Overall the technology use enhanced my level of learning in the course.” Pedagogy – effective faculty use of technology “The professor uses power point presentations to teach us the material, and it makes it easier to follow along and catch up on missed topics.”

Why did you choose this course as one that makes effective use of technology? Technology Interest – fulfilled students’ interest in using technology “This course played into both my love of technology and my major” Blackboard – frequently mentioned as tool employed in technology enhanced courses “All quizzes and notes are provided on BlackBoard”

Questions? Thank you! For more information please contact Danielle Mirliss at mirlisda@shu.edu