8th Sakai Conference4-7 December 2007 Newport Beach Programmatic Assessment Using Goal-Aware Activities in Sakai Daniel Tyger (SVSU) & Noah Botimer (UM)

2 Assessment Project Site Mid-MI Regional State Institution Saginaw Valley State University (since 1963) ~9500 Students Blackboard (BASIC) since 2000 Began piloting Sakai in SP ’06 Around 50 / 50 adjunct / full-time faculty

3 Project Goals Programmatic Data Sought –NCATE – accreditation-related, granular + summative data –Programmatic change indicators over time “Digital,” “Outcomes-based,” “portfolio” all used during the search for the perfect software that does everything OOTB for an indefinite period of time. Reporting data and digital student products both desired. (This is possible) Must be easy for faculty (and techs) and ‘natural’ for students. –“Matrices” were failing us here…

4 Primary Target Audiences 5 College of Education Programs –Reading MAT –Elementary Education (Undergraduate) –Elementary Education MAT (Graduate) –Early Childhood Education –Special Education Emotionally-Impaired Cognitively-Impaired Learning-Disabled Sociology Department (recent project)

5 Prerequisites Program faculty reach consensus: –Program goals (if not pre-determined by accrediting agency or national authority) –Benchmark assignments / activities that meet program goals –Earmark specific courses where these activities would be delivered –Rating scale and interpretation / application of it »Show assignment summary exampleassignment summary example »Show rating scales / definitionsrating scales / definitions

6 Program Worksites House template activities for the program’s courses All program instructors enrolled in site House program goals / goal set(s) Program rating scale source Editable by program coordinator (only!) Attach authoritative instructions, rubric to each activity Reports and activity summaries in Resources Requires established deadline dates before “injection” and reporting »Program worksiteProgram worksite »Show goal set examplegoal set example

7 Program Coordinator Role Consolidate proposed program changes for ITS by deadline –Changes in an activity’s linked goals –Activity name changes –Additional activities Maintain program worksite semester-semester –Accurate, generic activity instructions for each activity –Accurate attached rubric(s) for each activity Communicate to program faculty their responsibilities

8 Faculty Role 2 major responsibilities –Have students submit assignment(s) –Rate student performance against linked goals Training Issues? –Material online, , phone –Workshops Faculty impressions: “Easy” “No big deal” »Show training materials sitetraining materials site

9 Student Role Normal workflow and expectations –Makes sense to students –Something they do anyway Complete and submit assignments, as directed by faculty Non-punitive (not graded, per se) Student acclimates to concept of identifying benchmark assignments / learning outcomes

10 Sakai Tools Utilized Goal Management –Provides goal linking and rating for activities Assignments –Used where a student product is submitted, evaluated against a rubric, and rated on each linked goal. Course must be published. Data Points –Used when student achievement toward a goal is rated, but a submitted student product is not necessary.

11 Data Model / Strategy Mapped program enables automation for consistency and ease of use Comprehensive solution for outlined assessment scenarios Builds “Assessment Management” on top of Goal Management »Show Data ModelData Model

12 Program Tracking Setup Associate program with program worksite ID Associate assessments with a program Associate activities to assessments Extract activity template references from program site activities Associate activities to course sites / sections (“Deployment Context[s]”) »NCATE.adp shotsNCATE.adp shots

13 Deployment a.k.a. “Injection”

14 Deployment a.k.a. “Injection” Injection query (all activities  sites) Web services + helper scripts (Python) –Create all program course sites that don’t already exist, roster them –Associate course sites with program worksites –Verify needed tools (Assignments / Data Points) –Copy template activities into each site –Copy linked goals to each activity

15 Reporting Near semester close… –ITS runs audit of all program sites for existing activity / rating counts –Forward to program coordinator Contact delinquent faculty to rate their activities Granular Data –Every rating on every goal on every activity in a program can be collected across adjunct and full-time faculty Summative automated reports – next step BIRT Designer / Engine? »Show audit tool and exampleaudit toolexample »Show report examplereport example

16 Schedule of Tasks / Deadlines Agreement needed here Accountability, accuracy, punctuality needed

17 Longitudinal Concerns Archive snapshot of what happens each deployment period for future reference –Activities change over time –Linked goals may change –New activities added / removed over time –Comparisons / longitudinal studies complicated

18 Problems Communication & Coordination Faculty are very busy when programmatic changes are due Consistent interpretation of rating scale and application of that scale across many instructors Interpreting what NCATE actually wants? Faculty removing or altering activities after injection

19 Future “Assessment Management” (?) service as a contrib tool (Q1 2008) Program Mapper web interface Course Management integration BIRT integration for stock and ad-hoc assessment reports