LAKEFRONT SHERATON HOTEL COLUMBIA, MARYLAND 4-5 NOVEMBER 2014 CO-CHAIRS DAVID S. “HOOT” GIBSON, US AIR FORCE ACADEMY ELIZABETH “BETH” HAWTHORNE, UNION COUNTY COLLEGE, ACM EDUCATION BOARD Cyber Science Learning Outcomes Cyber Education Project Learning Outcomes Committee

Learning Outcomes Committee Charter The primary objective of this effort is to provide the cyber education community with useful guidance on developing undergraduate cyber science curricula. The Committee will build upon previous works to define related bodies of knowledge and seek diverse perspectives to build an interdisciplinary set of learning outcomes which broadly define cyber science education a the undergraduate level. The outcomes developed will also be used to guide the development of criteria for cyber science program accreditation. The Committee will develop learning outcomes which characterize the knowledge, skills, and abilities gained by students in an undergraduate cyber science program. Ultimately the work of the Committee should lead to a cyber science curricular guidance report formally endorsed by a professional society such as the Association for Computing Machinery (ACM).

Other Cyber Curriculum Development Work NSA/DHS Center for Academic Excellence (CAE) in Information Assurance/Cyber Defense (IA/CD) Knowledge Units (2014) NICE National Cybersecurity Workforce Framework version 2 (2014) - niccs.us-cert.gov/research/draft-national- cybersecurity-workforce-framework-version-20 niccs.us-cert.gov/research/draft-national- cybersecurity-workforce-framework-version-20 Department of Labor Cybersecurity Industry Model (2014) - models/cybersecurity.aspx models/cybersecurity.aspx Military Academy CYBER Education Working Group, Draft Body of Knowledge and Draft Outcomes, unpublished, 2014.

Other Cyber Curriculum Development Work cont’d ACM ITiCSE Working Group Papers ( )  An Exploration of the Current State of Information Assurance Education (2009) - dl.acm.org/citation.cfm?id= dl.acm.org/citation.cfm?id=  Towards Information Assurance (IA) Curricular Guidelines (2010) - dl.acm.org/citation.cfm?id= dl.acm.org/citation.cfm?id=  Information Assurance Education in Two- and Four-Year Institutions (2011) - dl.acm.org/citation.cfm?id= dl.acm.org/citation.cfm?id= ACM/IEEE-CS Curriculum Guidelines for Undergraduate Degree Programs in Computer Science: IAS KA (2013, p. 99) – cs2013.org cs2013.org ACM Toward Curricular Guidelines for Cybersecurity: Report of a Workshop on Cybersecurity Education and Training (2013) –

Other Cyber Curriculum Development Work cont’d CERT Software Assurance Curricula All Volumes CERT Software Assurance Curriculum Project Volume II: Undergraduate Course Outlines (2010) - resources.sei.cmu.edu/library/asset-view.cfm?assetID=9543 resources.sei.cmu.edu/library/asset-view.cfm?assetID=9543 CERT Software Assurance Curriculum Project Volume IV: Community College Education (2011) - resources.sei.cmu.edu/library/asset-view.cfm?assetID=10009 resources.sei.cmu.edu/library/asset-view.cfm?assetID=10009

Other Cyber Curriculum Development Work cont’d U.S. Department of Energy Essential Body of Knowledge (EBK): A Competency and Functional Framework For Cyber Security Workforce Development (2010) - energy.gov/sites/prod/files/cioprod/documents/DOE_EBK_June_201 0_Revision_clean_v01public.pdf energy.gov/sites/prod/files/cioprod/documents/DOE_EBK_June_201 0_Revision_clean_v01public.pdf DHS IT Security Essential Body of Knowledge (EBK): A Competency and Functional Framework for IT Security Workforce Development (2007) (ISC) 2 Common Body of Knowledge -

Why the Learning Outcomes Approach? Focus is on student achievement rather than on existing disciplines and courses (minimizes turf wars concerning who is in and who is out) Supports inclusive approach (it is easier to add many outcomes than many whole courses of study) Supports development of new courses in a new and evolving discipline Avoids traditional body of knowledge focus on topics and contact hours that can grow unbounded as new technologies emerge  What topics are eliminated to make room for the new?

Why the Learning Outcomes Approach? LOs: statements describe what students will be able to do as a result of learning LOs: students understand expectations and faculty can focus on student achievement LOs: specific measurement of student achievement having a specific minimum acceptable standard to pass (a threshold level)

Why the Learning Outcomes Approach? Active LOs are Active  action verbs describe what students should be able to do Aligned LOs can be Aligned  aligned with the rest of the curriculum; so LOs contribute to achievement of course outcomes, which in turn contribute to program outcomes Achievable LOs are Achievable  written at the threshold level for a pass, not aspirational Assessed LOs can be Assessed  possible to assess several learning outcomes with one assignment and can also be assessed informally, based on classroom tasks or discussions

Example Learning Outcomes Bloom’s Revised Taxonomy – acmccecc.org/BloomsTaxonomy/ acmccecc.org/BloomsTaxonomy/ CS2013 – Information Assurance and Security Knowledge Area (IAS KA) Foundational Concepts in Security Knowledge Unit LOs  Analyze the tradeoffs of balancing key security properties (Confidentiality, Integrity, and Availability)  Describe the concepts of risk, threats, vulnerabilities and attack vectors (including the fact that there is no such thing as perfect security).

Example Learning Outcomes cont’d CS2013 IAS KA cont’d Defensive Programming KU LOs  Explain why input validation and data sanitization is necessary in the face of adversarial control of the input channel.  Demonstrate using a high-level programming language how to prevent a race condition from occurring and how to handle an exception. Network Security KU LOs  Describe the architecture for public and private key cryptography and how public key infrastructure (PKI) supports network security.  Describe virtues and limitations of security technologies at each layer of the network stack.

Example Learning Outcomes cont’d CS2013 IAS KA cont’d Security Policy and Governance KU LOs  Describe how privacy protection policies run in conflict with security mechanisms  Identify the risks and benefits of outsourcing to the cloud Digital Forensics KU LOs  Describe the legal requirements for use of seized data.  Conduct data collection on a hard drive.

Activity What should students learn in cyber science? What common categories have emerged? The following slides show the technical and non- technical categories of answers placed on sticky notes.

“Technical Sticky Clumps” In no particular order Attack Cryptography Data Analysis Database Defense Ethical Hacking Forensics Hardware ICS/SCADA Math Mobile Networks Operating Systems Programming Reverse Engineering Secure Software Telecom

“Non-Technical Sticky Clumps” In no particular order Basic Principles Education Ethics Human Factors Lifecycle Organizations Policy, Governance and Law Privacy & Confidentiality Recovery Risk Management Strategy

Additional topics from follow-up meeting Threat landscape Psychological operations Business continuity, recovery Risk management Governance (triad with policy and law) International standards, policy, and law Resilience Intelligence Economics Embedded systems Systems design System safety Supply chain Artificial intelligence Linguistics History

Follow-on Questions How should cyber science outcomes differ from computer science outcomes? What is the best technical and non-technical mix? Who are the subject matter experts? What other questions should we ask? Do you currently have a program in “cyber” or are you considering developing one? What makes your cyber program different from other computing-based programs?

Learning Outcomes Development Timeline Early Dec 2014 – on-line organizational meeting  Define roles of committee members and topic area leads  Establish learning outcome format and repository  Establish dates and locations for future meetings Jan-Feb 2015 – on-line meetings  Discuss development of cyber science learning outcomes March 2014 – Face-to-face meeting (2 days)  Draft cyber science learning outcomes document June 2015 – Present work at CEP Workshop Fall 2015 – Broad review and comment on outcomes Spring 2016 – Publish learning outcomes report

Getting Involved in Outcome Development Sign-up at: Or contact Committee Co-chairs Beth or Hoot:  Beth Hawthorne:  David “Hoot” Gibson: