1. Qualtrics Proposal Gwen Gorzelsky, Executive Director, TILT
Matt Hickey, Chair, Committee on Teaching and Learning
UTFAB Presentation
11 April 2017

2. Background
Student Course Section Survey begun a number of years ago, at the specific request of ASCSU
ASCSU contributed to funding for over a decade
Paper forms
VERY labor-intensive to administer
Very labor-intensive to post process
Cause delay in reporting
Increasingly, comments from students: “Why isn’t this online?”

3. CSU Course Survey Improvement Goals
Appropriate use:
Student feedback helps instructors to revise course design, instructional materials, and teaching approaches
Students learn how peers see the learning environment created by specific a instructor in a particular course using an accessible tool
Expert design for intended uses
Questions suited to intended purpose; feedback aimed at improving the learning environment.
Design based on feedback from all stakeholder groups
Contributes to a culture of continuous improvement in learning and teaching
Green survey: Digital delivery

4. Institutional Aim: Excellence in teaching and learning
Task Force on Teaching Effectiveness
(TILT & UDTS)
Student Course Survey Redesign
(CoTL)

5. Building a culture of continuous improvement: Assessing Teaching Effectiveness Task Force
Year-long process: analyzed existing research on diverse approaches to assessing and mentoring teaching effectiveness
Report produced December 2015
Report circulated and discussed in multiple forums since:
Provost’s Advisory Student Success Committee
Council of Deans
Advisory Committee on Undergraduate Education
Executive Committee of Faculty Council
Full Faculty Council
CSU Fall Forum
TILT Summer Conference and Professional Development Institute
Faculty Council Committee on Teaching and Learning

6. Building a culture of continuous improvement: Course Survey redesign
Year-long process of feedback from faculty and students on existing tool, aims, concerns.
Analysis of research on student evaluation of teaching
Analysis of statistical use (and misuse) of data
Continual input from CoTL & ASCSU
Charge from FC:
Phase 1: common “core” questions
Phase 2: adaptable database of context-specific questions

7. Course Survey Shift in Focus: From Assessing Instructor to Improving the Learning Environment
Questions designed to obtain crucial perspectives available only from student feedback
Questions tailored to help instructors improve course design, instructional materials, and teaching approaches
Questions designed to encourage formative rather than summative uses of survey responses
Intent to administer selectively, with faculty participation, multiple times during the semester, to improve the learning environment
Bullet #1 Did the course learning environment . . .
Provide resources that helped me develop skills and/or understanding?
Motivate me to do my best work?
Provide regular, substantive feedback that helped me gauge my progress?
Bullet #2 Questions less about personal characteristics (e.g., about fairness, approachability, enthusiasm)
Questions more about behaviors that instructors can change (e.g., choosing course materials; encouraging students to study to understand, rather than to memorize; structuring collaborative study into course requirements
Questions focus instructors on specific, achievable goals (e.g., constructing a learning environment that requires students to regularly test knowledge levels, rather than allowing students to binge-study right before exams)
Bullet #3 Questions encourage department chairs to focus on instructors’ skill levels, rather than on personal characteristics
Questions therefore provide a clear focus for professional development when needed, rather than leading to questions about personality (e.g., “not approachable” or “not fair”)

8. In search of a platform: The Process
Reviewed several tools used internally
Followed threads on the EDUCAUSE CIO listserv
Reviewed tools used by peer institutions
Narrowed down to two products
AEFIS (new company, Unizin)
Qualtrics (established, used widely at CSU)
UTFAB Proposal 21 February, 2017

9. Usage About 220,000 course surveys submitted per year
Submitted by > 70% of CSU course sections
Used in every T&P document

10. Decision Matrix AEFIS Qualtrics The tool Much better The company
Reporting, data
Cost
Less costly
Overall
Much Better
UTFAB Proposal 21 February, 2017

11. Budget Proposal UTFAB funding request
$15k/yr. licensing for FY 18 (site license lowers cost)
$5k one-time implementation and configuration
$20k total
Will include annual costs in University Planning and Budgeting process for future funding in FY 19
Would fund ongoing costs, if approved
UTFAB Proposal 21 February, 2017

12. UTFAB Funding Criteria
1. Benefit as many students as possible
All CSU students likely to benefit
2. Ability to effectively utilize the fee
Your decision
3. Not funded by CFT
Correct
4. Adherence to budget and accountability
5. Potential for direct student use
6. Effort, thought & clarity in the plan
7. Quantitative usage data
8. Financial co-sponsorship
FTE, Operations, data analysis, integration, etc.
9. Central/distributed balance
10. Not part of other UTFAB projects
UTFAB Proposal 21 February, 2017

13. Discussion and Questions
Are most welcome
UTFAB Proposal 21 February, 2017