1. Assessing Distance Learners at Texas Tech University West Texas Assessment Conference Jennifer Hughes, M.S. Director, Office of Planning and Assessment Justin Louder, Ed.D. Assistant Vice Provost, TTU Worldwide eLearning Craig Morton, Ph.D. Associate Director, Office of Planning and Assessment October 21, 2015

2. Introduction - Jennifer Confluence of Institutional Activities  Reaffirmation activities that began in 2012 have strengthened Texas Tech’s commitment to assessing distance learners.  In Fall 2014, Texas Tech built a new infrastructure around distance education and online learning.  The university re-structured existing offices in Spring 2015 to support and build institutional effectiveness procedures.

3. Rationale for Assessing Distance Learning- Jennifer Compliance  Of the 95 SACSCOC standards, nearly 40% have some relevance to distance education  Scholarly literature has responded to the need to assess distance learners (Koper, 2015; Ribberbush; 2014; Smith & Macdonald; 2015)

4. Using Assessment Data for Improvement - Jennifer How can we use assessment as a tool for student learning improvement at Texas Tech?  We realized that we needed to develop a comprehensive distance learning assessment plan  This plan satisfies the institution’s need for compliance and for identifying areas of improvement

5. OLC Quality Scorecard – Justin OLC Quality Scorecard is a way for a program to determine the effectiveness of the online learning programs at the institution. Evaluates programs on the following categories: 1.Instructional Support 2.Technology Support 3.Course Development/Instructional Design 4.Course Structure 5.Teaching & Learning 6.Social and Student Engagement 7.Faculty Support 8.Student Support 9.Evaluations and Assessment Evaluation and Assessment: The program is assessed through an evaluation process that applies specific established standards. A variety of data are used to regularly and frequently evaluated program effectiveness to guide changes toward continual improvement. A process is in and followed for the assessment of stakeholder satisfaction with the online program.

6. Discussion of The Plan - Justin Texas Tech’s Distance Learning Assessment Plan is built around the following instruments:  CLA+  Online Senior Assessment (Locally Developed)  Modified Distance Education Learning Environments Survey  Priorities Survey for Online Learners  ETS Proficiency Profile

7. Longitudinal Emphasis of The Plan - Justin Longitudinal Assessment  Pilot testing began in Spring 2015 with the PSOL  DELES was administered in Fall 2015  CLA+ administration is underway  ETS Proficiency Profile will be administered in Fall 17/19

8. PSOL Pilot Results- Justin Piloted the Priorities Survey for Online Learners (PSOL) – Summer 2016 341 Respondents: 233 Female (68%) & 108 Male (32%) 32% were 19 to 24 28% were 25 to 34 41% graduate students 60% employed full time

9. PSOL Results - Justin Strengths Student assignments are clearly defined in the syllabus Instructional materials are appropriate for program content Assessment and evaluation procedures are clear and reasonable. Challenges Faculty are responsive to student needs Faculty provide timely feedback about student progress I am aware of whom to contact for questions about programs and services.

10. Using the Resulting Data for Improvement - Justin How will the results from this plan be used for improvement?  Demonstrate a need for more specialized student support services for distance learners  Identify areas of faculty development  Determine degree program offerings  Triangulate data from multiple sources

11. Real-Time Challenges to Assessing Distance Learners - Justin Real-Time Challenges  Cost  Identity Confirmation  Faculty Recruitment  Student Recruitment  Incentivization

12. Development of Matrix - Jennifer To describe how each instrument aligns with each student competency, a matrix was developed.  Distribution of matrix  Discussion of matrix

13. Distance Learning Assessment Matrix - Jennifer CLA+ (Collegiate Learning Assessment Critical Thinking SkillsEvery third year beginning 2015-2016 100General Learning Direct OSA (Online Senior Assessment) Core Curriculum CompetenciesEvery other year beginning Spring 2017 Total Population Curricular Learning Direct DELES-M (Modified Distance Education Learning Environments Survey) Program DevelopmentEnd of every fall semester beginning Fall 2015 Total Population SatisfactionIndirect PSOL (Priorities Survey for Online Learners) – Noel Levitz Student SatisfactionEnd of every spring semester beginning Spring 2015 Total Population SatisfactionIndirect ETS Proficiency ProfileCritical Thinking, Reading, Writing, and Mathematics. Fall 2017 and Fall 2019 200General Learning Direct National Survey of Student Enagement Program DevelopmentEvery odd yearApproximate ly 1500 Student Engagement Behavior and Attitudes Indirect SACSCOC Standards Responses Student Learning, Program Development, and Administrative Oversight Consistent with reaffrimation cycles Qualititative Review ComplianceIndirect

14. Distance Learning Assessment Matrix - Jennifer MethodDescription CLA+ (Collegiate Learning AssessmentCLA+ is an innovative collegiate-level assessment designed explicitly for the twenty-first-century student. Unlike most standardized tests, which measure the ability to memorize numbers, dates, and facts, CLA+ measures critical-thinking skills, which employers have overwhelmingly stated matter more to them than a particular major or GPA. The skills CLA+ tests for include analysis and problem solving, scientific and quantitative reasoning, critical reading and evaluation, and critiquing an argument. OSA (Online Senior Assessment)The OSA included a total of 32 knowledge questions (i.e., questions that can be answered either correctly or incorrectly) from the following core curriculum areas: Humanities (4 questions), Multicultural (7 questions), Mathematics (5 questions), Natural Sciences (6 questions), and Social and Behavioral Sciences (10 questions). DELES-M (Modified Distance Education Learning Environments Survey) The Distance Education Learning Environments Survey (DELES) is a psychosocial learning environment survey designed specifically to measure college and university distance education learning environments. The DELES was developed in 2003. The DELES was designed to measure students' perceptions of their distance education social environment through six scales: Instructor Support, Student Interaction and Collaboration, Personal Relevance, Authentic Learning, Active Learning, and Student Autonomy. PSOL (Priorities Survey for Online Learners) – Noel Levitz The first survey of its kind designed specifically for online students, the PSOL can tell you how satisfied your students are and what issues are really important to them including the areas of: Academic services, Enrollment services, Institutional perceptions, Instructional services, and Student services. ETS Proficiency ProfileThe ETS® Proficiency Profile assesses four core skill areas — critical thinking, reading, writing and mathematics — in a single, convenient test that the Voluntary System of Accountability (VSA) has selected as a gauge of general education outcomes. Plus, get greater insight into your students’ writing skills with a direct and authentic measure of their ability to construct a response by adding an essay to the ETS Proficiency Profile assessment. National Survey of Student EnagementSurvey items onThe College Student Report represent empirically confirmed "good practices" in undergraduate education. That is, they reflect behaviors by students and institutions that are associated with desired outcomes of college. NSSE doesn’t assess student learning directly, but survey results point to areas where colleges and universities are performing well and aspects of the undergraduate experience that could be improved. SACSCOC Standards ResponsesSACSCOC requires that all standards that have distance education components be addressed adequately. This includes information ons tudent services (CR 2.10), Facilities (CS 3.11.3), etc. However, there are a few standards with infromation about distance that is most applicable: Comprehensive Standard 3.4.7: Consortia relationships/contractual agreements, omprehensive Standard 3.4.8: Noncredit to credit, Comprehensive Standard 3.13.2: Collaborative academic arrangements, Comprehensive Standard 3.13.4.a: Review of distance learning programs, and Federal Requirement 4.8: Distance and correspondence education.

15. Preparing Degree Program Assessment Plans - Jennifer  As we implement this effort, we are also developing modality-specific degree program assessment plans  Fifth Year Interim will soon be upon us  The timing is right for our offices to partner together down a new road

16. Audience Discussion Questions - Jennifer  Do you have a comprehensive assessment plan for your distance learners? How did you develop it? Have you implemented it?  How do you assess distance learners separately from campus- based learners?

17. Contact Information  Ms. Jennifer Hughes, Jennifer.s.hughes@ttu.eduJennifer.s.hughes@ttu.edu  Dr. Justin Louder, Justin.louder@ttu.eduJustin.louder@ttu.edu  Dr. Craig Morton, craig.morton@ttu.educraig.morton@ttu.edu Please contact us if you would like to further discuss distance learning assessment.