Int. Conf. e-Learning 2011 Project Management and Assessment of Distributed Teams in an Online Capstone Masters-Level Information Technology Course Charles Tappert and Allen Stix Pace University, New York

Int. Conf. e-Learning 2011 Five Previous Conference Papers on Real-World Student Projects 1. Integrating Real-World Projects for Actual Customers into Capstone Courses (E-Learn 2002) 2. Security-Related Real-World Projects (ISECON 2004) 3. Interplay of Student Projects and Student-Faculty Research (E-Learn 2007) 4. Pedagogical Issues in Managing Information Technology Projects Conducted by Geographically Distributed Student Teams (SITE 2009) 5. Assessment of Student Work on Geographically Distributed Information Technology Project Teams (E-Learn 2009)

Int. Conf. e-Learning 2011 Real-World Student Projects Conducted in capstone courses for 10 years Student teams build real-world computer information systems for actual customers Project systems serve the community internal university community at Pace greater university community external non-profit local community

Real-World Student Projects (cont) Real-world projects are a stellar learning experience for students Win-win situation for all Students Customers Instructors and other involved faculty School of CSIS University Int. Conf. e-Learning 2011

Migrate to Online Format Migrated from traditional face-to-face format to online format in Fall 2006 To be progressive Technology for online courses adequate Online preferred by employed students – no scheduling conflicts & no commuting To expand the population of students beyond the greater NYC area Int. Conf. e-Learning 2011

Challenges of Online Format Uncertainties of how traditional course methods port to the online environment and what new methods might be required Teams lacking co-presence require higher level of organizational and process skills No weekly classroom meetings as safety net for teams’ interaction and functioning Int. Conf. e-Learning 2011

Team Projects – Categories Int. Conf. e-Learning 2011

Team Projects – Sources Int. Conf. e-Learning 2011

Team Projects – Publication Types Int. Conf. e-Learning 2011

Journal 2010 Team Projects – Spring 2011 Course website “Projects” page Fall 2009 Int. Conf. e-Learning 2011

Team Website Project title and description Project members and customers All deliverables posted Weekly status reports Midterm & final presentation slides User manual Technical paper Int. Conf. e-Learning 2011

Team Project – Example Website Personality Assessment from Handwriting Project

Journal 2010 Team Project – Example Website

Team Projects – Example Systems Handwriting Forgery Quiz System Rare Coin Grading System Biometric Systems Projects ( ) Keystroke biometric system Stylometry system Biometric product investigation Freeware – face & voice Licensed products – face & fingerprint Int. Conf. e-Learning 2011

Keystroke & Stylometry Systems The 2008 U.S. federal Higher Education Opportunity Act (HEOA) requires institutions of higher learning to make greater access control efforts to assure that students of record are those actually accessing the systems and taking exams in online courses by adopting identification technologies as they become more ubiquitous To meet these needs, keystroke and stylometry biometrics were investigated towards developing a robust system to authenticate online test takers Performance statistics on keystroke, stylometry, and combined keystroke-stylometry systems were obtained on data from 40 test-taking university students Int. Conf. e-Learning 2011

Keystroke Biometric System Keystroke biometric systems measure typing characteristics believed to be unique to an individual and difficult to duplicate Most previous work on passwords – two commercial products Keystroke biometric is appealing for online student verification not intrusive to computer users inexpensive – only hardware required is a computer with keyboard text continues to be entered for potential repeated checking after an initial authentication phase (referred to as dynamic verification) Our keystroke system developed over last 6-7 years is unique collects raw keystroke data over the Internet focuses on long-text input where sufficient keystroke data are available to permit the use of powerful statistical feature measurements Int. Conf. e-Learning 2011

Keystroke Biometric System Spring 2011 Results The best equal-error-rate performance on the keystroke system was 0.55% which is an improvement over earlier reported results on this system Receiver Operating Characteristic (ROC) curves: False Reject Rate (FRR) vs False Accept Rate (FAR) Experiments 1 and 2: Open systems – trained and tested on different students Experiment 3: Closed system – trained and tested on all students Int. Conf. e-Learning 2011

Stylometry Biometric System Stylometry is the study of determining authorship from the linguistic styles of the authors Traditionally, it has been used to attribute authorship to anonymous or disputed literary documents More recently, computer-based communication and digital documents have been the focus of research, sometimes with the goal of identifying perpetrators or other malicious behavior We are just beginning to apply stylometry to verify online student test takers, early results poor Int. Conf. e-Learning 2011

Stylometry Biometric System The performance of the stylometry system a poor 33% EER, indicating that stylometry is not suitable for text lengths of short-answer tests ( words per question) Receiver Operating Characteristic (ROC) curves: False Reject Rate (FRR) vs False Accept Rate (FAR) Experiments 1 and 2: Open systems – trained and tested on different students Experiment 3: Closed system – trained and tested on all students Int. Conf. e-Learning 2011

Biometric Product Investigation Freeware systems (usually 30-day trial) – fall 2010 project Face recognition KeyLemon (KeyLemon) & FaceSDK (Luxand) Voice recognition VoiceCipher (VoiceLatch) & bio-iVault/iWallet (myBiodentity) Inexpensive licensed products – spring 2011 project Face recognition SDK (Neurotechnology) Fingerprint recognition SDK (Neurotechnology) Results - freeware and cheap licensed systems equally poor Eigenface algorithm (coded in 2003 project) worked better Recent info exposes poor accuracy of biometric products Int. Conf. e-Learning 2011

Simple Eigenface Algorithm coded in 2003 Project Int. Conf. e-Learning 2011

? Query Face DB Face Recognition: System Int. Conf. e-Learning 2011

Biometric Identification Tom Cruise in Minority Report, 2002 Int. Conf. e-Learning 2011

Inspirational Portrait of Individuality Int. Conf. e-Learning 2011

Face Recognition: National Security Int. Conf. e-Learning 2011

Speaker Individuality: “My name is …” Int. Conf. e-Learning 2011

“My name is” from Two Different Speakers Speaker Individuality Int. Conf. e-Learning 2011

“My name is” divided into seven sound units. Speaker Individuality Int. Conf. e-Learning 2011

biomouse Fingerprint scanner Digital Camera LCD Pen tablet Microphone Multi-modality Biometric Authentication Embeded & Hybrid User Verification system System that requires user verification Int. Conf. e-Learning 2011

Issues/Solutions Stemming from Scattered Teams Project stakeholder communication Issue – communication gets difficult For example, scattered team members more likely to feel isolated and want to communicate directly with instructor or customer Solution Communication between team and instructor/customer must be through team leader distribution lists for whole class and for each team Project team leaders must be local to facilitate communication/meetings with instructor and customers Course website provides central source of course information Blackboard discussion forum for each project (see below) Int. Conf. e-Learning 2011

Issues/Solutions Stemming from Scattered Teams (cont) How to handle quizzes, deliverables, etc. Issue – classroom meetings not available Solution – use Blackboard educational software Quizzes Collecting digital deliverables Discussion forums Forum for archiving instructor Forum for student introductions Forum for textbook and other course material Forum for each team project Int. Conf. e-Learning 2011

Issues/Solutions Stemming from Scattered Teams (cont) Provide some face-to-face interaction Issue – no weekly classroom meetings Solution – three classroom meetings for local students/customers 1. Near beginning of course 1. Face-to-face introductions, nature of course, specifics of course, student team project meetings 2. Midterm 1. Project status presentations 3. End of semester 1. Final project presentations Int. Conf. e-Learning 2011

Current Assessment of Online Students Individual quizzes (20%) Blackboard educational software system Team initial assignment (10%) Students learn to function as a team Team project midterm checkpoint (20%) Team project final checkpoint (20%) Team technical paper (30%) Strong emphasis on projects No midterm/final exams (used in two-semester course) Int. Conf. e-Learning 2011

Team Member Self and Peer Evaluations Issue – lack of classroom meetings makes it difficult to determine individual team members’ contribution to the project work Peer evaluations critical for distributed teams Minimal team member/customer contact Minimal team member/instructor contact Literature indicates Various granularity levels in peer evaluations Some automated systems reported Int. Conf. e-Learning 2011

Team Member Self and Peer Evaluations Three times during the semester After initial assignment to learn the process At the midterm checkpoint At the final end-of-semester checkpoint Process for a graded team event First assign a team grade Adjust individual grades up/down based on self/peer, customer, and instructor evaluations Int. Conf. e-Learning 2011

Example Team Peer Evaluation and Grade Chart (4 member team) Team Member Eval 1Eval 2Eval 3Eval 4SummaryGrade 1+= ==–– – – –79 3–=+––83 4==–+=85 Average=====85 +/- 2% for each summary +/- sign, showing only peer evaluations. Int. Conf. e-Learning 2011

Pedagogical Course Evaluations Issue – lack of classroom meetings makes it difficult for instructor to determine relative value of the course methodologies Solution – semester-end survey (Survey Monkey) Procedures/methods that worked well, or did not work well, and why Int. Conf. e-Learning 2011

Pedagogical Customer Evaluations Issue – instructor is often not aware of the quality of team-customers interactions Solution – semester-end survey Obtain student feedback on customer interaction Were customer requirements clear? Was amount of contact/interaction adequate? Was help on the project work appropriate? Int. Conf. e-Learning 2011

Something New in Compared Projects to Reality Game Shows Capitalize on student interest in TV game shows Like Survivor and The Apprentice Common with project teams & game show groups Work toward common goals with division of responsibilities Acquire and share new knowledge Harness skills of different teammates Adjust to different personalities Exhibit initiative without disruptiveness Differences between projects & game shows Survivor “voted out” & The Apprentice “you’re fired” Game show participants compete against each other Int. Conf. e-Learning 2011

Conclusions  Over five year’s experience in face-to- face mode  Over four year’s experience in online mode  Techniques for managing and assessing distributed teams have been successful and they continue to evolve Int. Conf. e-Learning 2011