2008 FAEIS Triennial Evaluation The purpose of the FAEIS Triennial Evaluation is to conduct a comprehensive assessment of FAEIS once every three years in order to improve the usability of the information provided by FAEIS, monitor progress during the three-year grant cycle and plan for the future.

Objectives o Describe selected demographic and other variables that have potential association with FAEIS use. o Determine the level of usefulness of FAEIS data resources and applications. o Determine the level of satisfaction with selected components of FAEIS. o Determine levels of adoption of FAEIS and factors that explain the levels of adoption. o Determine the relationship of selected demographic and usage variables (involvement, use, experience, primary job title) with perceived usefulness and satisfaction with FAEIS.

Methodology o FAEIS Team developed first draft of questions o Peer Panel and USDA partners reviewed and made suggestions to draft questions o Peer Panel members were used to pilot test the on- line survey instrument. Changes made accordingly. o Web survey administered by the VT Center for Survey Research o 1,145 addresses from FAEIS users used as survey sample group. Reduced to 1,120 by request of sample members. o Follow-up reminders sent to sample members o 396 respondents completed surveys for a response rate of 35 %

Question 1: My professional affiliation is best described as ? 2007 Percent of Total (N=308) 2008 Percent of Total (N=285) 2009 Percent of Total (N=397) Staff Faculty Department Head College Associate/Assistant Dean College Dean or VP University-level Administration Other College-level Administration5.0 Government Non-Govt., Organization0.0 Institutional Research3.5 No answer1-- Analysis – fewer staff respondents in ‘09; greater number of faculty and administrator respondents in 2009 compared to 2007 & 2008.

# of Respondents by Institution Type 1862 – – – 6 PNLG – 109 Private – 21 Did not report -12 n=396 Question Respondents by Institution Type

n=384 Question 3 Indicate which of the following organizations, if any, is your primary affiliation. Organization Percentage of Responses APS10.7 % AASCARR9.6 % AEA/ARD1.3 % BOHS8.6 % SAF3.4 % AAVMC0.8 % CAFCS6.5 % NAUFRP4.7 % APLU16.1 % USDA1.8 % Other10.2 % Does not apply26.3 %

Question 45 Years in your current position? N=383

Questions 43 & 44 Demographics Female55.70 % Male44.30 % Asian1.60 % African American/ Black8.80 % American Indian / Alaskan Native0.50 % Hispanic0.80 % Native Hawaiian/ other Pacific Islander 0.50 % White85.20 % Other2.60 %

Three-Year Trends: Question 4 : My total experience using FAEIS is ? Analysis – Users with > 3 yrs experience is trending upward while the other two groups are trending down.

Question 5 My Involvement with FAEIS Includes: (select as many as apply; all possible combinations displayed) N= Results

Three-Year Trends: Question 5: Involvement with FAEIS of Survey Respondents ( “ none” and “ no answer” responses eliminated in ’07 & ‘08 ) Analysis – Data Entry and Product Developer survey respondents trending downward; Users of FAEIS products is trending upward.

Three-Year Trends: Question 6: How often have you used FAEIS in the past 12 months ? Analysis –The more frequent use categories (3-4 times & >4 times) are trending downward. The less frequent use categories (not at all & 1-2 times) are trending upward.

Three-Year Trends: Question 7: “My level of adoption (use) of FAEIS is best described as:” (no answer – eliminated from analysis in ‘07 & ‘08) Analysis – Only “unaware” and “use it but not sure of long-term use” are trending upwards. “Do not use” trending downward.

“ My level of adoption (use) of FAEIS is best described as:” (no answer – eliminated from analysis in ‘07 & ‘08) 2007 Mean 2008 Mean 2009 Mean Data Entry Users Only2.49 (N=159) 2.30 (N=152) 2.32 (N=157) All Other Users3.16 (N=117) 3.24 (N=125) 2.93 (N=240) Total2.78 (N=276) 2.73 (N=277) 2.69 (N=396) 1=unaware 2=aware of it but do not use it 3=Use it but not sure of long term use 4=Use it and it is integral to my job 5=Integral to my job and expect long term use 6=So important as to never be without FAEIS Analysis – Average level of adoption of FAEIS decreased in ’09 for non-data entry respondents

Question 21 Who are the users of FAEIS data at your institution/organization/agency ? n=391

Question 9 Is FAEIS collecting data from the institutions which you consider your peers? n=395 n=271

Question 10 If you answered no to: “Is FAEIS collecting data from the institutions which you consider your peers?,” Which Institutions are missing ? o Private schools with agriculture programs o Family and Consumer Sciences programs at NLG

Question 11 Do a sufficient number of institutions report data to allow for you to have a meaningful and helpful peer analysis? n=389 n=210

Three-Year Trends: Question 13: Describe your level of satisfaction with FAEIS components. (i.e., newsletter, help desk, data entry, report builder, instructions) (1=not satisfied; 5 = very satisfied) Analysis – Higher and lower satisfaction ratings are trending downward. The neutral rating (3) is trending upward.

Question 14 How useful to you are the following FAEIS products and resources? Highest Ratings: o CIP lists o Newsletter o Web site Lowest Ratings: o Teaching Awards / Workshops website o FAEIS Data CD n=396

Question 14 h Are there any other FAEIS products or resources you would like to add to the list ? Some Responses: o Salary data for placement data o Trend data such as degree programs that are changing over a 10 year cycle o Summary of IPEDS major trends that influence food and agricultural sciences o Similar data from 1994 land grant universities obtained from sister institutions.

Question 8 Does the CIP system adequately allow you to classify your instructional programs and/or address your reporting needs? n=392

Question 12 What aspect of FAEIS is most important to you ? Most Common Responses: o Student Enrollment data o Comparisons between institutions o Trends in enrollment, degrees awarded o Faculty salaries

Question 15 For which of the following purposes have you used FAEIS data and/or reports? ( Please check all that apply) N= Results Analysis – Benchmarking and trend analysis are the two most common uses of FAEIS data.

Question 16 Please rate the usefulness of the data resources in FAEIS (for student enrollment, degrees awarded and placement, disciplines, gender, ethnicity, faculty salaries, etc.) (1 = not useful, 5 = very useful) 2009 Results N=397

Three-Year Trends: Question 16 : Please rate the usefulness of the data resources in FAEIS (for student enrollment, degrees awarded and placement, disciplines, gender, ethnicity, faculty salaries, etc.). Analysis – The higher usefulness scores (4, 5) are trending upward. The lower usefulness scores (1, 2) are stable & trending downward.

Question 17 Please indicate how much confidence you have in the following FAEIS data resources. Table 15. Respondent Confidence in Types of FAEIS Data Not Confident At All Not Very ConfidentNeutral Somewhat Confident Very Confident Do Not Know Total college enrollment Student enrollment by CIP Student demographics Degrees awarded by CIP Student placement by CIP Faculty headcounts Faculty demographics Faculty salaries International programs st 3rd 2nd = Least confident= Most confident n=386

Question 18 Is FAEIS the primary place you go to seek higher education data and other information relating to the food and agricultural sciences, natural resources and conservation, and family and consumer sciences/human sciences ? n=391

Question 19 Do you use any of the following databases ? Other: o CRIS, o NAUFRP, o USDA databases n=280

Question 20 To what extent does FAEIS add value to other information sources that are available ? n=384

Question 24 Have you contacted the FAEIS Help Desk for assistance ? n=388

Question 25 How would you rate the support you received from the Help Desk ? n=153

Question 26 If you have contacted the FAEIS Help desk, how could its operation be improved ? (n=146) 22 % - No improvement needed. 1 % - Increase hours of operation 0.3 % - Improve response time 12.4 % - Do not know 1.8 % - Other suggestions: o Help Desk staff is limited by the lack of thorough, well- edited, accurate and consistent documentation, web screen content and communications. o Information provided by the Help Desk is not consistent from time-to-time. o Develop databases more thoroughly o Help Desk employees need better training o Clear understanding of the parameters of the data being entered o More understanding of the difficulties institutions are having entering the data

Question 27 How would you rate the ease of navigation of the FAEIS web site? n=390

Question 28 Have you used customized FAEIS data reports produced by the FAEIS Report Builder application ? n=393

Question 29 Have you personally used the FAEIS Report Builder application ? n=390

Question 30 How well does the FAEIS Report Builder meet your needs ? n=71

Question 31 How easy is it to use the FAEIS Report Builder ? n=70

Question 32 Are the instructions for the FAEIS Report Builder clear ? n=71

Question 33 Would training on how to use the FAEIS Report Builder be useful to you? n=71

Question 34 Do you enter FAEIS data, or are you responsible for data entered into FAEIS ? n=389

Question 35 How would you rate your ability to obtain the following data in the format required by FAEIS ? Enrollments by degree level, discipline, gender, and ethnicity Degrees awarded by degree level, discipline, gender, and ethnicity N=262N=260

Question 35 How would you rate your ability to obtain the following data in the format required by FAEIS? Faculty by discipline, rank, gender, and ethnicity Placement by degree level into various CIP disciplines and work categories N=257N=260

Question 36 Are data on placement of students collected at your institution ? N=134

Question 37 At your institution, from where do you acquire the data for the various surveys ? N=

Question 38 What time of year would be best to enter the following types of FAEIS data ? N=

Question 9 What data entry method do you use for the following surveys ? N=

Question 41 Are the instructions clear for FAEIS data entry methods that you use ? N=240 If answered no, which instructions need to be improved? o Would prefer questionnaire by mail o Navigation on screen a pain o Difficult to discern how to move from in-progress to completed o Finishing an entry is difficult o Difficult to navigate and decide when to save o Not sure what majors you want to be collected o Submission instructions are vague o Time frames for info requested needs to be stated clearly

Question 22 How can FAEIS be more useful to your institution or be improved ? Most Common Responses: o Data input is cumbersome, needs to be easier, bothersome. o Need more information about what FAEIS is and what it can do for us. o I need to use it more. o Concerned about the accuracy and credibility of the data. o Complaints about CIP codes

Question 23 Please share any comments about when FAEIS was helpful to you or about when you had a problem with FAEIS. Most common “Helpful” Responses: o Staff was helpful. (21) o Useful for faculty salary comparisons.(4) o Helpful in grant boilerplates (1) o Student diversity by CIP info (1) o Dropping / adding programs (1) o Growth of programs, trends in majors (2) o Long-term planning (1) Most Common “Problem” Responses: o Data not dependable or accurate.(6) o Data entry not user friendly (5) o Report builder not user friendly. (2) o Poor job of marketing / facilitating use of FAEIS (1)

SUMMARY o The more years using FAEIS the greater the satisfaction with FAEIS o Greatest satisfaction of FAEIS data and its use: enrollment, institutional benchmarking, trend analysis, salary comparisons o Greatest satisfaction of FAEIS products: CIP code list, e-newsletter, website o FAEIS report builder—good, but room for improvement o Most confident in FAEIS data: college enrollment, faculty headcount, student/faculty demographics, degrees awarded: least confident: international/student placement o Most users believe FAEIS adds-value & is used twice as often as any other similar database o Most users have not contacted help desk; but of those that have--- 90% rated it excellent or good. o Users get FAEIS data from many places—Depts/Colleges/IR/HR

Question 47 Please provide any suggestions you have regarding this survey and the individual survey items. N=383 Most common and pertinent responses: o Does not know FAEIS; Does not use FAEIS; Lacks experience with FAEIS; Does not see benefit of FAEIS o Affiliation question (Q1) was strange o Should have indicated how survey results will be used o Limit survey to users that have at least 1 year of experience with FAEIS