FAIR Best Paper: Using Technology to Efficiently and Effectively Gather Information from Recent Alumni and Employers May 2010 Association of Institutional Research 50 th Annual Forum

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research2 Alumni and Employer Surveys Presenters  Dr. James Coraggio, Director of Academic Effectiveness and Assessment Other Authors Not Present  Maggie Tymms, Assessment Coordinator for Academic Programs  Nicole Carr, Former Assessment Coordinator for Academic Programs

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research3 SPC Background  SPC, established in 1927, is the oldest 2-year college in Florida  First Community College in Florida to offer 4 year degrees (2002)  9 Campuses throughout the county  FTE: 16,966.8 (LD), 1,676.7 (UD)  Opening Fall 2009 credit enrollment: 29,560  Annual headcount: 61,414

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research4 Institutional Effectiveness “Institutional Effectiveness is the integrated, systematic, explicit, and documented processes of measuring performance against the SPC mission for purposes of continuous improvement of academic programs, administrative services, and educational services offered by the college.” “Closing the Loop”

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research5 Performance Improvement  From Compliance to Performance Improvement

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research6 Agenda  Student Assessment Points  Purpose of Follow-up Surveys  Alumni/Employer Survey Process  Dissemination of Survey Results and Reports

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research7 Student Assessment Points Students surveyed multiple times

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research8 Purpose of Follow-up Surveys Recent Alumni Survey was designed to:  Gain insight into alumni’s perceptions of the quality of general education and career preparation  Determine the current activities of former students such as employment information and educational status

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research9 Purpose of Follow-up Surveys Employer Survey was designed to:  Gain insight into an employer’s perceptions of hired SPC graduates in the areas of key general education competencies and foundational skills  Provide the employee’s major job responsibilities and wage information, as well as employer’s willingness to hire another SPC graduate  Identify employer’s willingness to support various college activities

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research10 Purpose of Follow-up Surveys State Requirements  Survey data are also collected for state required annual Placement & Follow-up Report (FETPIP: Florida Education Training Placement Information Program) and supplemental information to meet workforce criteria

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research11 Alumni/Employer Survey Process Previous model (Before 2005)  Labor intensive  Manually tabulated paper surveys  Telephone interviews conducted using overtime hours

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research12 Alumni/Employer Survey Process Issues with previous model (Before 2005)  Costly survey process  Declining college budget  Increasing demand for accountability  Outdated contact information  Students over-surveyed  Declining response rates  Conducted in isolation

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research13 Alumni/Employer Survey Process Goals of new survey process A.Improve data collection process B.Enhance quality of survey forms C.Employ industry best practices D.Include discipline specific information E.Increase response rates

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research14 A. Improve Data Collection Process Data Collection Process  Eliminated telephone calls  Automated the process (Remark Office OMR and merge)  Employed online survey submission (Zoomerang TM )  Obtained accurate addresses  Used multiple points of contact

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research15 A. Improve Data Collection Process Survey Schedule  Recent Alumni Survey  Graduates surveyed six months after the end of their last term (Three times per year)  Employer Survey  Employer of graduates surveyed at the end of academic year (Once per year)

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research16 A. Improve Data Collection Process First Second Third First mailing Second mailing 2 weeks Recent Alumni Survey Multiple Points of Contact 2 weeks

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research17 B. Enhance Quality Of Survey Forms Improved Survey Forms  Better ‘look and feel’ of survey format  Developed scannable paper forms  Used online branching models to reduce survey length  Designed items to gather only pertinent data

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research18 C. Employ Industry Best Practices Don Dillman’s Five Elements  Respondent-friendly questionnaire  Up to five contacts with the respondent  Inclusion of stamped envelope  Personalized correspondence  A token financial incentive that is sent with the survey request

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research19 Online Survey: and Form

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research20 Paper Survey: Letter and Form

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research21 C. Include Program Specific Information Program specific information  Met with Deans/Program Directors  Discussed requirements for program accreditation and timelines  Included their necessary program specific items

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research22 D. Increase Response Rates Create awareness and convey importance  Involved program faculty in the process  Requested faculty assistance to inform students  Included a flyer with graduation materials  Coordinated with programs that hand- carry surveys to local employers

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research23 Alumni Survey Response Rates  Summer 2007

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research24 Alumni Survey Response Rates  Fall 2007

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research25 Alumni Survey Response Rates  Spring 2008

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research26 Alumni Survey Response Rates  Summer 2008

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research27 Alumni Survey Response Rates  Fall 2008

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research28 Alumni Survey Response Rates  Spring 2009

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research29 Alumni Survey Response Rates  Summer 2009* *Summer 2009 in progress

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research30 Alumni Survey Response Rates *Summer 2009 in progress

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research31 Alumni Survey Response Rates Source of Responses

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research32 Alumni Survey Response Rates *Summer 2009 in progress

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research33 Alumni Survey Response Rates Overall Response Rate *Summer 2009 in progress

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research34 Alumni Survey Response Rates Graduate Year Surveyed Number Surveyed Number Responded Percent Responded Number of Online Responses Percent of Responses Online ,1811, % % ,7011, % % , % % , % % ,4171, %1006.6% Note: For , only partial Summer 2009 results have been received.

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research35 Response Rates - Employer Other Responses Recent Alumni Surveys Sent Employer Surveys Sent Is a 44% Employer Response Rate Good? COE Responses Permission Required Responses Permission Not Required

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research36 Disseminating Information  College wide access to reports   Program Specific Reports  Published on the Educational Outcomes website 

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research37 Disseminating Information

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research38 Disseminating Information

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research39 Disseminating Information

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research40 Program Specific Reports

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research41 Recent Improvements  Survey invitations from program administrators (signatures)  Additional program specific reports (i.e., baccalaureate programs and PSAV)  Improved oversight (and QC) of printing functions

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research42 Lessons Learned  Do not expect that students will remember their ID#s  Administer the survey soon after the end of the session (six months)  Use multiple points of contact (two weeks)  Find the most recent contact information  Partner with programs to acquire up-to-date information

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research43 Lessons Learned  Make students aware of the importance of the survey process (survey announcements)  Provide contact updates to college data systems (PeopleSoft, Alumni Foundation)  Monitor results and make changes as necessary  Use automated processes for a quicker turn around of results and more accurate data

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research44 Future Direction  Our ultimate goal is to provide Program Administrators ‘timely’, ‘relevant’ ‘accurate’, and ‘interpretable’ information regarding their programs and their performance outcomes  Multi-measure ‘Dashboard’ style reports (Business Intelligence/Performance Measurement)

AIR 50 th Annual Forum May 2010 May 31, 2010Association of Institutional Research45 Questions/Next Steps

FAIR Best Paper: Using Technology to Efficiently and Effectively Gather Information from Recent Alumni and Employers May 2010 Association of Institutional Research 50 th Annual Forum