1. How Technologically Literate are EMCC Students?

2. Technological Literacy
A Comparison of 2012 and 2015 Assessments
SAAC Report 11/25/15

3. Purpose and Methodology
Fall 2011: Technological Literacy assessment rubric developed by SAAC
Designed to assess student technological literacy across the College
Spring 2012: Technological Literacy assessment implemented
Spring 2015: Reassessed with same rubric

4. Purpose and Methodology
Seven common “deliverable components” were assessed:
Data management
References
Assignment content
Communication
Layout
Embedded objects
Conventions

5. What was Assessed?
Data managing: was the respondent’s name properly displayed, was the course identified, and was the file saved in the correct format?
References: were the appropriate number of references used, were all references listed in APA or MLA style, and were the references embedded into the document?
Assignment content: was there evidence of effective research, did the student achieve the purpose of the assignment, and was the correct audience targeted?
Communication: was the assignment successfully transmitted, and does the student provide evidence supporting the retrievability of the file
Layout: did headings/subheadings reflect a logical structure, was there a consistent visual theme, were appropriate font color and size used, and were animations appropriately used?
Embedded objects: did the deliverable contain the following types of objects: pictures, video, audio files, animation, hyperlinks, and other (per instructor)?
Conventions: phrases are of appropriate length for the medium, punctuation is appropriate, grammar usage is appropriate, and spelling is accurate?

6. 2012-2015 Participant Information
Prefixes 8 7
Sections 15 14
Instructors 11 6
Students
237
234
Class Level
0 Dev Ed, 22% 100 level, 78% 200 level
1 Dev Ed, 28% 100 level, 68% 200 level
Sophomores
19% Sophomores
47% sophomores
Freshmen
42% New Freshmen
20% New Freshmen
Lowest Areas
Embedded Objects, References

7. Overall Proficiency (scored at 3 or 4)

8. Overall Differences in the Means
Aggregate score of the Technology Literacy assessment was high in both years
2012: 3.80 (on a 4-point scale)*
2015: 3.60 (on a 4-point scale)*
* statistically significant level

9. COMPARING NEW FRESHMEN to SOPHOMORES 2015 RESULTS
Mean scores for Sophomores were only statistically significantly different than Freshmen in Data Management.
Freshmen: 3.83 (on a 4-point scale)*
Sophomores: 3.98 (on a 4-point scale)*
* statistically significant level

10. Analysis - Limitations
Cannot extrapolate the results to represent the college population due to small sample size
Differences in scoring methodologies between 2012 and 2015.
Faculty did not identify specific deliverable components missing in 2012 (just if the total number was sufficient).
This was corrected in 2015 but may compromise the ability to accurately compare results between the two assessments
Proportional changes between new freshmen and sophomores further compromises the ability to accurately compare results between the two years
With only one instructor responsible for almost half of all 2015 assessments, the final results cannot be extrapolated to be representative of the college at large
Anecdotally three instructors (representing more than half the students sampled) indicated their grading rigor increased between 2012 and 2015

11. Questions and Suggestions
Developmental history of students using technology: is Ppt really appropriate assessment platform?
Does Ppt reflect what is really needed for technological literacy? What about LMS?
Workforce reality: Ppt is widespread
Importance of establishing baseline assessment of Tech knowledge coming in, as well as going out
Working on assumption that students have a basic level of tech understanding (e.g. Canvas)
(eLearning advisory committee working on creating a course for students to become literate in Canvas)
International technological standards can be used for future reference?
Challenge of new generation with reading issues

12. Questions and Suggestions
Our students are hitting ceiling – high enough standards? Gear assessment to higher industry standards
Importance of transference of skills
Fall 2017: SAAC needs to take a deep look at appropriate assessment, industry standards, international standards – incorporate 21st century technology skills
Can we take a look at where/when CIS105 is offered? What about involving them in the assessment
Importance of faculty collaboration, use team leaders, cross curricular heterogeneity, etc.
Future discussion of CIS105 involvement
What about students taking a tech literacy survey? Eliminate variables and bias. Some type of standardized test, each class could participate
Open source assessments?
Continue discussion of certain curriculums/areas having primary responsibility of assessment in their acknowledged discipline

13. Questions and Suggestions
Ideas for increasing faculty involvement?
Letter from president
More interaction, easier pathway to integration into what is already an overloaded curriculum
Department specific ideas for each assessment
Establish workshops for faculty to work on future assessments
Possible to pare down assessments to be discipline-specific related
For establishing criterion, ASU requirements may be used for developing gen ed assessments
Search schedule for Classes>sorted by Gen Ed designation; also district curriculum site

14. Questions and Suggestions
Our students are hitting ceiling – high enough standards? Gear assessment to higher industry standards
Importance of transference of skills
Fall 2017: SAAC needs to take a deep look at appropriate assessment, industry standards, international standards – incorporate 21st century technology skills
Can we take a look at where/when CIS105 is offered? What about involving them in the assessment
Importance of faculty collaboration, use team leaders, cross curricular heterogeneity, etc.
Future discussion of CIS105 involvement
What about students taking a tech literacy survey? Eliminate variables and bias. Some type of standardized test, each class could participate
Open source assessments?

15. Suggestions from SAAC and Leadership Council 11/26 Meeting
Get feedback from Classroom Conversations (12/2)
Need to get more faculty involved
Review the rigor of the assessment (is it too easy?)
Inform the faculty that references, embedded objects were lowest areas of proficiency for two cycles in a row.
Do we enable sloppy behavior?
Work with Instructional Computing faculty members for creating a tutorial video for how to embed objects into student work.
Work with Writing Center to integrate appropriate format for references across the curriculum
More of a interactive tutorial?
Separate assignment for students before tech literacy assessment?
Writing Center remind faculty every semester that reference tutorials are available
Include Ppt citations guidelines – video specific to this
Continue for SAAC co-chairs to visit divisional meetings to encourage participation