1. Assessment of Teaching and Learning with Technology at Seton Hall University
Janet Easterling, Institutional Research
Danielle Mirliss, Senior Instructional Designer
Shayle Abelkop, Instructional Designer
Copyright Abelkop, S., Easterling, E., and Mirliss, D. (2007). This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.

2. Why Mobile Computing
“Seton Hall University’s Mobile Computing Program is designed to break down the barriers of space and time and allow all our students to fully participate in our scholarly community and achieve high levels of performance.”
Mobile Computing Mission Statement
This has been our main impetus for assessment – we have been at this for some time – program in place since 1996

3. Mobile Computing Timeline
1994 – 1996: Development of SHU Strategic Plan
1995 – 1997: Mobile Computing Pilot Project with small cohorts of students in Business School
1997: Mobile Computing Pilot Project expands to 300 incoming Freshman
1998: Mobile Computing Program launched for all incoming Freshman
2001: Mobile Computing Program expanded to select graduate programs; campus wireless network implemented
2002: SHU graduates first mobile class
2004: Tablet PC Pilot Project
Poll the audience – who is responsible for doing assessment on their campus? Who is actually engaged in assessment of technology? Have audience describe where they are and their interest in this topic?

4. Mobile Computing @ SHU Innovative academic program involving
three components:
Access: The University licenses student use of a laptop computer as part of tuition and fees.
Curricular Integration: The University encourages and supports faculty use of technology in teaching and learning.
Network and Support Services: The University provides the infrastructure that enables effective use of technology in teaching and learning.
This is how we have viewed our mobile computing program – the inspiration for our assessment efforts. From the T&L perspective the curricular integration is very important.

5. Mobile Computing Assessment
Began in 1996
Student Survey
Historically focused on:
Overall Satisfaction
Use of the Technology
Impact on Effective Teaching Practice
Renamed Teaching and Learning with Technology Survey
Emphasis on Student Use and Effective Integration of Technology
Make sure to mention survey fatigue – mixing the technical with the pedagogy . Talk about importance of team - stakeholders

6. Early Efforts/Challenges
In an effort to assess everything the survey grew to monstrous proportions
Felt a need to document clearly positive outcomes of efforts to infuse
Then compelled to monitor positives
Added questions but also kept all the questions…
Slide on what to assess – cannot ask everything

7. Better but not there yet…
Survey questions were narrowed to emphasize teaching and learning
Exploratory approach to achieve better questions– open-ended/focus groups
Smaller number, better questions
Application of results to inform practice
Slide on what to assess – cannot ask everything

8. Where we want to be…
Ask questions that are meaningful to all stakeholders (even students!!)
Craft a survey each year based on needs in a changing context
Disseminate information to campus and beyond
Include additional stakeholders to drive questions
Other people are driving what we want to look at

9. Results from along the way…
Quantitative – student responses on usage, gains, preferences & benefits
Qualitative – opinions, details of perceived benefits and barriers
Other people are driving what we want to look at

10. T & Learning with Technology Students
Quantitative highlights
Hours each week on-line Q3
Usage of IT to Q4
IT preferences in courses Q5
IT improves collaboration, feedback, control? Q6
IT improves my learning Q7
IT feature most valuable Q8
Gains d.t. on-line Bb/web Q12
Opps 7prins qual learning Q13
Janet

11. Learning with Technology Student Survey
administered April 2006 to a random sample of freshmen and juniors
oversampling for males and for students identifying as non-white
253 responses, overall response rate 30%
Higher rates for Women, Residents, FR
Lower rates for Men, Non-White, Commuting Students, JR

12. 1 –. Hours each week on-line Q3 How many hours per week would you estimate you spend…
Most frequent response
6-10 hours / week on-line (www/ /..) for academic work avg 11 hours
~1 in 2 students (a little under) specify
more than 10 hours a week
~1 in 5 students (a little over)
5 or fewer hours a week
~1 in 10 students (a little over)
21 or more hours a week
45% more than 10 hours (11% 21+)
23% 5 hours or less ( 4% zero)

13. 1 –. Hours each week on-line Q3 How many hours per week would you estimate you spend ….
Most frequent response
6-10 hours / week on-line (www/ /..) for non-academic work avg 13 hours
~1 in 2 students (a little over) specify
more than 10 hours a week
~1 in 4 students
5 or fewer hours a week
~1 in 4 students (a little over)
21 or more hours a week
55% more than 10 hours (27% 21+ hours)
25% 5 hours or less (11% zero)

14. ~3 in 5 students specify 21 or more ! ~1 in 4 students 10 or fewer
1 –. Hours each week on-line Q3 How many hours per week would you estimate you spend..
Summed responses for hours / week on-line (www/ /..) in total avg 23 hours
~3 in 5 students specify or more !
~1 in 4 students or fewer
~1 in 10 students or more !!
60% more than 20 hours (12% 41+ hours)
25% 10 hours or less ( 2% 5 or less)

15. 2 – Use of IT for Academic work Q4 On average how often do you use Information Technology to.. Daily/Weekly/Monthly/Rarely/Never
Frequency of use of IT for academic work
to complete course assignments 77% D/W
e-resources from Library web % D/W
e-resources from other sites % D/W
(other than a library web site)

16. 2 – Use of IT for Academic work Q4 On average how often do you use Information Technology to.. Daily/Weekly/Monthly/Rarely/Never
Freq of use of IT for academic work
to work in teams
during class 62% D/W/M
outside of class % D/W/M
(to complete course assignments)

17. 3 – Student prefs for use of tech in courses Q5 I prefer taking courses that …
first choice, courses that
use a moderate level of technology % ( , several ppt presentations, some on-line activities/content)
however, almost tied !
use technology extensively % (class lecture notes on-line, ppt, computer simulations, streaming video)
third choice
use limited technology % (professor , limited ppt use)

18. 4 – Impact of IT Q6 Compared to courses that use little or not technology, the use of technology … (agreement)
use of IT allows me to
Take greater control of my course activities planning, apportioning time, awareness of success/failure % (42% strongly agree, 37% agree)
use of IT results in
More prompt feedback from my instructor % (40% strongly agree, 45% agree)
use of IT helps me
Better communicate and collaborate with my classmates % (38% strongly agree, 42% agree)

19. has Improved my Learning 80% (strongly agree or agree)
5 –IT Improves Learning? Q7 The use of technology in my courses has improved my learning… (agreement)
use of IT in my Courses
has Improved my Learning % (strongly agree or agree)
21% strongly agree
59% agree
15% neutral
3% disagree
2% strongly disagree

20. benefit from using technology 68% convenience / saves time
6 – IT Most valuable benefit Q8 Which of the following benefits from using technology in your classes are most valuable to you…
single most valuable
benefit from using technology
68% convenience / saves time
19% helps manage class activities
9% improves my learning
3% other 2% none

21. Course selected by the student
7 – Perceived gains from on-line (Bb/web) use Q12 As you answer the following questions, think about one particular course from among those you are taking this semester that, in your opinion, makes effective use of technology to support teaching and learning… In this course, what was the effect of having the following on-line features (available through Blackboard or your professor’s web site)?
Course selected by the student
as one that makes effective use ..
(student reasons for selecting –
Danielle Mirliss will present these)

22. Course selected by the student
7 – Perceived gains from on-line (Bb/web) use Q12 As you answer the following questions, think about one particular course from among those you are taking this semester that, in your opinion, makes effective use of technology to support teaching and learning… In this course, what was the effect of having the following on-line features (available through Blackboard or your professor’s web site)?
Course selected by the student
as one that makes effective use ..
59% Mostly Undergraduate – FR/SO
39% Mostly Undergraduate – JR/SR

23. In course selected by the student
7 – Perceived gains from on-line (Bb/web) use Q12 As you answer the following questions, think about one particular course from among those you are taking this semester that, in your opinion, makes effective use of technology to support teaching and learning… In this course, what was the effect of having the following on-line features (available through Blackboard or your professor’s web site)?
In course selected by the student
as one that makes effective use ..
Improved Learning %responses
Online readings & links to materials 67%
Online discussion board 47%
Sharing materials with students 45%
Improved Management of Class Activities
Online syllabus %
Ability to turn in assignments on-line 73%

24. In course selected by the student
7 – Perceived gains from on-line (Bb/web) use Q12 As you answer the following questions, think about one particular course from among those you are taking this semester that, in your opinion, makes effective use of technology to support teaching and learning… In this course, what was the effect of having the following on-line features (available through Blackboard or your professor’s web site)?
In course selected by the student
as one that makes effective use ..
Positive Experience 0/1 Mean
Online syllabus
Online readings & links .86
Online turn in assignments .79
Online sharing materials .70
Online get grades, comms .66
Online track your grades .64
Online discussion board .63

25. In course selected by the student
8 – Opportunity for 7-principles learning env’s Q13 As you answer the following questions, think about one particular course from among those you are taking this semester that, in your opinion, makes effective use of technology to support teaching and learning… In this course, how often were you able to do or experience the following)?
In course selected by the student
how often able to do or experience ..
Student Faculty Contact %often {very}
Have contact with professors %
on course content
Diverse ways of learning % often {very}
Explore course materials %
Demonstrate what I have learned %
.. in different ways

26. In course selected by the student
8 – Opportunity for 7-principles learning env’s Q13 As you answer the following questions, think about one particular course from among those you are taking this semester that, in your opinion, makes effective use of technology to support teaching and learning… In this course, how often were you able to do or experience the following)?
In course selected by the student
how often able to do or experience ..
Cooperation among students %often {very}
Spend time with other students on %
Feedback
Get prompt feedback on performance 73%
Observe & record my own progress 57%
Time on Task
Edit and improve my projects / assign. 60%

27. 8 – Opportunity for 7-principles learning env’s Q13 As you answer the following questions, think about one particular course from among those you are taking this semester that, in your opinion, makes effective use of technology to support teaching and learning… In this course, how often were you able to do or experience the following)?
Does positive experience with related on-line activity relate to frequency of opportunity for high quality learning environments?
Cooperation among students %often {very} Q13b
Spend time with other students on %
Positive experience with on-line Q12
Online readings & links
Online sharing materials
Online discussion board mean correlation with 13b

28. 9 – Strengths, Challenges Quantitative data
More gen’l observations come back to..
Strengths .. results
s-f contact (as in prior years)
s-s interaction (newly revealed)
Challenges .. data
? serious responses .. (given incentive)
? generalizable (given response %)

29. 10 –Teaching & Learning with Technology survey assessment continued
Next ..
Reviews of qualitative - Danielle & Shayle
helps ..
hinders ..
effective use ..
+Focus Groups to probe, explore
- “effective uses” of technology – examps ..
- gains from interactions – which online resources..
Survey Qs next cycle that probe effective use
in terms meaningful to student

30. Qualitative Responses

31. Technology helps learning by…
Providing anytime, anywhere access
Organizing information
Facilitating communication among students/professors
Broadening research/resource materials
Efficient methods of submitting assignments
Making access equitable
Having projectors in class
Exercising technical skills

32. What students said…
“Giving us access to certain resources that benefit our learning experience as a whole”
“Giving students an outlet to learn valuable skills”
“Making it easier to contact professors and fellow students for help, notes, etc.”
“Technology is always available and accessible when I need it, whether I want to take notes in class with my laptop or use resources available through the school’s homepage”
“With all the information that is now provided via the internet, the technology we have guides us to easy access for attaining this information.”

33. Technology prevents learning by…
Providing distractions
Technical issues
Too much reliance
Facilitating plagiarism
Less personal contact w/ students
“At Seton Hall, technology can prevent learning because during lectures some students may be too busy using instant messenger instead of paying attention to the professor.”

34. What students said…
“Being a distraction for those who do not know when to separate from it at needed times.”
“Having access to cheating methods.”
“Technology can prevent students from learning because they are able to surf the net while in class as well as talk on AIM.”
“Many students bring their laptops to classes and become distracted by them because it gives them something to do besides paying attention.”

35. Why did you choose this course as one that makes effective use of technology?
Assignments – ability to complete and submit assignments electronically
“This course not only provides me with a syllabus online but with other assignments that are to be done online.”
Collaboration – use of technology to communicate and co-author assignments
“I thought it allowed me to work better with my classmates.”

36. Why did you choose this course as one that makes effective use of technology?
Convenience – enables students to keep track of work and updated materials
“Kept me up to date with my course grades and assignments.”
Feedback – provides a way for faculty to give feedback on assignments
“Many of our assignments are turned in online via our discussion board and we are given feedback on certain assignments from our professor as well.”

37. Why did you choose this course as one that makes effective use of technology?
Frequency – course uses technology more than other courses
“Because technology is used for every aspect of the class: assignments, quizzes, lectures, discussion board, etc.”
Information Literacy – ability to access resources from library and Internet
“We use the library database every day.”

38. Why did you choose this course as one that makes effective use of technology?
Learning – contributed to students’ understanding of course material
“Overall the technology use enhanced my level of learning in the course.”
Pedagogy – effective faculty use of technology
“The professor uses power point presentations to teach us the material, and it makes it easier to follow along and catch up on missed topics.”

39. Why did you choose this course as one that makes effective use of technology?
Technology Interest – fulfilled students’ interest in using technology
“This course played into both my love of technology and my major”
Blackboard – frequently mentioned as tool employed in technology enhanced courses
“All quizzes and notes are provided on BlackBoard”

40. Questions?
Thank you!
For more information please contact Danielle Mirliss at