How Technologically Literate are EMCC Students?

Technological Literacy A Comparison of 2012 and 2015 Assessments SAAC Report 11/25/15

Purpose and Methodology Fall 2011: Technological Literacy assessment rubric developed by SAAC Designed to assess student technological literacy across the College Spring 2012: Technological Literacy assessment implemented Spring 2015: Reassessed with same rubric

Purpose and Methodology Seven common “deliverable components” were assessed: Data management References Assignment content Communication Layout Embedded objects Conventions

What was Assessed? Data managing: was the respondent’s name properly displayed, was the course identified, and was the file saved in the correct format? References: were the appropriate number of references used, were all references listed in APA or MLA style, and were the references embedded into the document? Assignment content: was there evidence of effective research, did the student achieve the purpose of the assignment, and was the correct audience targeted? Communication: was the assignment successfully transmitted, and does the student provide evidence supporting the retrievability of the file Layout: did headings/subheadings reflect a logical structure, was there a consistent visual theme, were appropriate font color and size used, and were animations appropriately used? Embedded objects: did the deliverable contain the following types of objects: pictures, video, audio files, animation, hyperlinks, and other (per instructor)? Conventions: phrases are of appropriate length for the medium, punctuation is appropriate, grammar usage is appropriate, and spelling is accurate?

2012-2015 Participant Information Prefixes 8 7 Sections 15 14 Instructors 11 6 Students 237 234 Class Level 0 Dev Ed, 22% 100 level, 78% 200 level 1 Dev Ed, 28% 100 level, 68% 200 level Sophomores 19% Sophomores 47% sophomores Freshmen 42% New Freshmen 20% New Freshmen Lowest Areas Embedded Objects, References

Overall Proficiency (scored at 3 or 4)

Overall Differences in the Means Aggregate score of the Technology Literacy assessment was high in both years 2012: 3.80 (on a 4-point scale)* 2015: 3.60 (on a 4-point scale)* * statistically significant level

COMPARING NEW FRESHMEN to SOPHOMORES 2015 RESULTS Mean scores for Sophomores were only statistically significantly different than Freshmen in Data Management. Freshmen: 3.83 (on a 4-point scale)* Sophomores: 3.98 (on a 4-point scale)* * statistically significant level

Analysis - Limitations Cannot extrapolate the results to represent the college population due to small sample size Differences in scoring methodologies between 2012 and 2015. Faculty did not identify specific deliverable components missing in 2012 (just if the total number was sufficient). This was corrected in 2015 but may compromise the ability to accurately compare results between the two assessments Proportional changes between new freshmen and sophomores further compromises the ability to accurately compare results between the two years With only one instructor responsible for almost half of all 2015 assessments, the final results cannot be extrapolated to be representative of the college at large Anecdotally three instructors (representing more than half the students sampled) indicated their grading rigor increased between 2012 and 2015

Questions and Suggestions Developmental history of students using technology: is Ppt really appropriate assessment platform? Does Ppt reflect what is really needed for technological literacy? What about LMS? Workforce reality: Ppt is widespread Importance of establishing baseline assessment of Tech knowledge coming in, as well as going out Working on assumption that students have a basic level of tech understanding (e.g. Canvas) (eLearning advisory committee working on creating a course for students to become literate in Canvas) International technological standards can be used for future reference? Challenge of new generation with reading issues

Questions and Suggestions Our students are hitting ceiling – high enough standards? Gear assessment to higher industry standards Importance of transference of skills Fall 2017: SAAC needs to take a deep look at appropriate assessment, industry standards, international standards – incorporate 21st century technology skills Can we take a look at where/when CIS105 is offered? What about involving them in the assessment Importance of faculty collaboration, use team leaders, cross curricular heterogeneity, etc. Future discussion of CIS105 involvement What about students taking a tech literacy survey? Eliminate variables and bias. Some type of standardized test, each class could participate Open source assessments? Continue discussion of certain curriculums/areas having primary responsibility of assessment in their acknowledged discipline

Questions and Suggestions Ideas for increasing faculty involvement? Letter from president More interaction, easier pathway to integration into what is already an overloaded curriculum Department specific ideas for each assessment Establish workshops for faculty to work on future assessments Possible to pare down assessments to be discipline-specific related For establishing criterion, ASU requirements may be used for developing gen ed assessments Search schedule for Classes>sorted by Gen Ed designation; also district curriculum site

Questions and Suggestions Our students are hitting ceiling – high enough standards? Gear assessment to higher industry standards Importance of transference of skills Fall 2017: SAAC needs to take a deep look at appropriate assessment, industry standards, international standards – incorporate 21st century technology skills Can we take a look at where/when CIS105 is offered? What about involving them in the assessment Importance of faculty collaboration, use team leaders, cross curricular heterogeneity, etc. Future discussion of CIS105 involvement What about students taking a tech literacy survey? Eliminate variables and bias. Some type of standardized test, each class could participate Open source assessments?

Suggestions from SAAC and Leadership Council 11/26 Meeting Get feedback from Classroom Conversations (12/2) Need to get more faculty involved Review the rigor of the assessment (is it too easy?) Inform the faculty that references, embedded objects were lowest areas of proficiency for two cycles in a row. Do we enable sloppy behavior? Work with Instructional Computing faculty members for creating a tutorial video for how to embed objects into student work. Work with Writing Center to integrate appropriate format for references across the curriculum More of a interactive tutorial? Separate assignment for students before tech literacy assessment? Writing Center remind faculty every semester that reference tutorials are available Include Ppt citations guidelines – video specific to this Continue for SAAC co-chairs to visit divisional meetings to encourage participation