Accreditation Process under OBE {POs} August, 2019

Outcomes at the Heart of the Accreditation Process

Over time, Learning Outcomes have come to Play a Crucial Role in the Accreditation Processes {Almost} the World Over.

Of Course, OBE is not only for Accreditation It is about “Quality of Education” Curriculum OBE Curriculum & Teaching/Learning Assessment

Qu. What are the Roles of these two components? Curriculum & Teaching/Learning Assessment Qu. What are the Roles of these two components?

A Bit of History

Yes, it is needed, But what is it? Confusion Yes, it is needed, But what is it?

What does the term "Outcome-Based Education" really mean? Outcome-Based Education: Critical Issues and Answers by William G Spady What does the term "Outcome-Based Education" really mean?

Outcome-Based Education means clearly focusing and organizing every-thing in an educational system around what is essential for all students to he able to do** successfully at the end of their learning experiences. This means starting with a clear picture of what is important for students to he able to do, then organizing curriculum, instruction, and assessment to make sure this learning ultimately happens.

In Engineering, OBE Leads to: Defining of the Graduate Profile Through Learning Outcomes Defining “Good” How do we do it?

A Couple of more Questions: Qu.1. Who Defines the Learning Outcomes? Ans. The Stakeholders of that Program. Then, Qu.2. Who are the Stakeholders? Ans. In Engineering- Industry and Academia

Key Terms Accreditation Criteria Program Educational Objectives – PEO Course Outcomes – CO Program Outcomes – PO Assessment Evaluation

Accreditation is a Judgment on the Program- Acceptable/ Not Acceptable To make consistent and objective we need criteria to base the judgment. It should not depend on who are the Evaluators. Has to be Objective & Consistent Criteria

How to make the judgment? Criteria-Rule book Evidence on Record Criteria is your rule book. You examine evidence only in the light of these criteria and then arrive at a judgment. Why this slide? Because, your judgment has to be on the basis of criteria and on the evidence available. Not on any perceptions. This brings in uniformity. Does not depend upon the individual evaluator. As dictated by the Accreditation Criteria and Evidence on record

Role of the two criteria Program Curriculum PO Program Outcomes Curriculum and the Teaching/ Learning processes are the basis on which the program is built Attainment of POs indicates that the job is well done PC and PO are completely different in their roles.

One Question (?) In an Institution all the inputs - Infrastructure, students, faculty, curriculum- are Good. Q. Will the graduates be “NOT GOOD*?”? Ans. “May be- May be Not:” We just don’t know! In OBE, we determine Twin question marks. In OBE we answer both the questions. What do we understand by “Good” and then determine How Good?

Clearly, Outcomes have to be (and Are) Measurable Term in an introductory slide “make sure”

Evidence "Not a particle of evidence, Pip," said Mr. Jaggers, shaking his head. "Take nothing on its looks; take everything on evidence. There's no better rule." What is true in life is also true for accreditation.

So, while writing their report: For Evaluators So, while writing their report: The Evaluators could always state as: “Based on the following evidence presented and evaluated by us the following conclusions can be arrived at {…}”

SAR-Evaluations – Major Factors Vision and Mission & PEO Curriculum & Teaching- Learning {Program Outcomes (PO) Course Outcomes (CO)} Consistency and Interactions Amongst These- Matrices Correspondence matrices

We Examine PEO- very briefly for the sake of completeness. It answers the question: Why has the program been created in the first place In the context of OBE these are the ones that really matter.

Short Definitions of CO, PO Program Outcomes- POs These are knowledge and skill sets that the graduates have at the time of graduations** Course Outcomes- COs Again, these are knowledge and skills that are attained by the graduates in a course {Could be a part of a PO} COs are mapped onto POs { CO-PO matrix}

Qu. Why are the NBA’s POs , What they are? Not arbitrary

WA – Graduate Attributes and NBA- Program Outcomes Washington Accord Attributes NBA Program Outcomes. 1.Engineering knowledge, Apply knowledge of mathematics, science, engineering fundamentals and an engineering specialization to the solution of complex engineering problems. 1.Engineering knowledge, Apply the knowledge of mathematics, science, engineering fundamentals, and engg. specialization to the solution of complex engineering problems 2. Problem Analysis, Identify, formulate, research literature and analyze complex engineering problems reaching substantiated conclusions using first principles of mathematics, natural sciences and engineering sciences 2. Problem Analysis, Identify, formulate, research literature, and analyze engineering problems to arrive at substantiated conclusions using first principles of mathematics, natural, and engineering sciences. WA defined the profile of the graduates by articulating precisely a series of attributes, and NBA’s Pos are derived from these attributes.

Clearly, Outcomes have to be Measurable

NBA-Learning Outcomes 1. Engineering Knowledge, 2. Problem Analysis 3. Design/development of solutions, 4. Conduct investigations of complex Problems, 5. Modern tool usage, 6. The engineer and society, 7. Environment and sustainability, 8. Ethics, 9. Individual and team work, 10. Communication, 11. Project management and finance, 12. Life-long learning Criteria 2 26

Program Outcomes - POs Have to be Understood Some are difficult to Attain Also, difficult to Assess and Evaluate {Today’s discussion is about this Aspect} We examine all these aspects.

1. Engineering knowledge: Apply the knowledge of mathematics, science, engineering fundamentals, and engg. specialization to the solution of complex engineering problems. 2. Problem analysis: Identify, formulate, research literature, and analyze engineering problems to arrive at substantiated conclusions using first principles of mathematics, natural, and engineering sciences. 3. Design/development of solutions: Design solutions for complex engineering problems and design system components, processes to meet the specifications with consideration for the public health and safety, and the cultural, societal, and environmental considerations. 4. Conduct investigations of complex problems: Use research-based knowledge including design of experiments, analysis and interpretation of data, and synthesis of the information to provide valid conclusions.

Two Terms- Assessment & Evaluation

For Evaluators Assessment It is one or more processes that identify, collect, and prepare data to evaluate the achievement of Program Outcomes and program educational objectives Important for both, SAR and the Evaluators

For Evaluators Evaluation These are processes for interpreting the data and evidence accumulated through assessment practices. Evaluation determines the extent to which POs or PEOs are being achieved, and results in decisions and actions to improve the program as also for accreditation.

Attainment of PO1 to PO4

Why place these four POs in one Basket? The Statements show that one part {That of (complex) Engineering Problem CEP} is common to all. Though, individually each PO deals with a different aspect of CEP. Recognizing this commonality makes the discussion easier. So, there will be many points that are to all; and then the differences.

The different aspects of CEP Application of Mathematics and sciences- PO1 Identify and Research to Solve – PO 2 Design and Development of Solutions – PO 3 Conducting Investigations and Research PO 4

We need to be able to Determine if the course does have such problems in its Assessment

Sample Indicators of Complex Engineering Problems-CEP {Some Hints for the Evaluators} 1. Problems not the kind generally encountered at the ends of text book chapters. (These often test if the contents of the chapter have been understood) 2. These are problems that have not been completely framed and leave at least a few* choices for the student to make. 3.Problems may require use of laws of physics, or bring in some mathematical tools in which the problem can be framed. Making choices is not straightforward. First, to determine; what are the options available – open ended. It requires some experience, understanding,, engineering knowledge ability weigh, often conflicting factors- that is why it becomes a complex engineering problem

Choices – Not Easy {Hints Continued} Qu. What are the available Choices? {Open Ended} Requires: Experience, Engineering Knowledge, Understanding, Ability to define and manage constraints that are Applicable, and manage these. These make it a Complex Engineering Problem

Tier -1 and Tier-2 For tier 1 , (2) could read as- only skeletal details provided in the problem, and student is required make out the rest. Such a problem could require amongst others defining constraints in terms of power, cost, weight, life span, different engineering choices, etc. For such problems, there are, in all probability multiple valid solutions. Difference between Tier-2 and Tier 1, in this respect, is in the degree of complexity and numbers

Difficult vs. Complex

Another Take on CEP {from CEAB, Canada}

1. It must require the application of in-depth knowledge According to the CEAB, a complex engineering problem is defined by the following characteristics: 1. It must require the application of in-depth knowledge 2. It must satisfy at least one of the following additional characteristics: • involves wide-ranging or conflicting issues • has no obvious solution such that originality is required • involves infrequently encountered issues • is outside accepted standards and codes • involves diverse stakeholders and needs • is posed at a high-level with many components or sub-problems

Factors We Examine Assessment: { Information and Evidence for Evaluation of attainment} Evaluation: {From the evidence making judgments on attainment levels} We examine both the factors from procedural and, evidence interpretation aspects

The First Step Towards Assessment: { We recognize that POs are attained through the COs. So we determine the corresponding set COs. These COs in turn lead us to courses - the places where we find information and evidences} This step is pictured in the next slide. We will need the CO-PO matrix for this. It is there in the SAR

Look for Evidence In that Course! The Sequence From Which CO? From Which Course? A Given PO Look for Evidence In that Course!

Mapping from the CO-PO matrix {from SAR} Set of {COs} & {Associated set of Courses}

Every Course Leads to Some Outcomes Every Course Leads to Some Outcomes. All the courses must cover the stated list of outcomes. One way of verifying this to prepare a match matrix as shown below. In the table below * could also be a number- typically in (0,1) indicating level of attainment. Course PO PO 1 PO 2 PO 9 EE 111 * EE 212 HS 101 There is considerable difference between planning of this matrix, and evaluation of it’s attainment. 46

Every Course Leads to Some Outcomes Every Course Leads to Some Outcomes. All the courses together must cover all the POs (and PSOs). For a course we map the COs to POs through the CO-PO matrix shown below. Assume that it is for a course EE111 CO PO PO 1 PO 2 PO 9 CO 1 * CO 2 CO n There is considerable difference between planning of this matrix, and evaluation of it’s attainment. 47

To Begin with, an Evaluator Asks: Qu. How “Good” are the COs? Determining Attainment of POs To Begin with, an Evaluator Asks: Qu. How “Good” are the COs?

Are the COs well defined? CO Analysis- How well do the CO statements match with the PO statements? Very often a part of a PO is embedded in the CO. Else, we reason on the degree of match. Continued CO Statement – Compare – PO statement

Answer to this question could be in terms of degree of match such as: (a) Excellent/ 4 (b) Good / 3 (c) Fair /2 (d) Little /1 {To be recorded for all the COs. Examples follow}

Now, To Determine the Attainment of a CO:

Assessment Procedure Step 1- Collect information from question papers, assignments, tutorials, answer scripts, etc {Mostly in course files} Step 2- SAR might also furnish projects {mini and major} and Lab work as evidence towards attainment The two steps together give assessment.

Step 3: Analysis of questions in examinations, tutorials, assignments, etc*. will point to those questions that are devised to evaluate attainment of COs. Again, How Good? Step 4: An examination of the relevant answer scripts along with the extent of relevance of the questions with the COs will reveal the actual level of attainment. {Steps 3 and 4 are judged by you as the domain expert. The CO Attainment levels need to be determined and recorded in your notes. Some illustrative examples will follow } Avichal*

CO - Attainment From a single CO to a PO Let x be the relevance of a CO to a PO A question {under consideration} has relevance to the CO is, judged by you as y. Attainment of the CO in question by the students (in exam, test, assignment,…) is z. Then contribution of the CO to the PO is: xyz All assessments are done by the Evaluators Scales for x, y, z are all in the range (1-4)

Normalization & Aggregation (xyz)takes values from 1 to 64. Normalize by dividing by 64 so that the range is 1/64 to 1. Let this be denoted by [xyz] For a PO: Attainment level would be: PO(Attainment) =  [xyz]; sum over all related COs

PO Attainment Establish Attainment levels for all the COs applicable for that course by Inspection of the evidence available for the purpose. Then, as per the CO – PO matrix, created for the purpose, map these results into PO attainment levels. { Cumulate as indicated} Continue this with selected set of courses so that attainment levels are established for all POs- PO1-PO4.

So, while writing their report: For Evaluators So, while writing their report: The Evaluators could always state as: “Based on the following evidence presented and evaluated by us the following conclusions can be arrived at {…}”

Why is PO 1 needed? Modern Engineering Practice requires a very good Understanding of Mathematics, Physics, and Basic Engineering Sciences. Such an understanding helps in tackling problems encountered in professional practice as well as development tasks that have to be carried out. PO 1 essentially proves these abilities.

Our Job as Evaluators is to ascertain whether PO 1 has been attained in the Program

Illustrative Example A Course on Basic Electric Circuits has this as : CO1 {as per SAR}: Ability to apply laws of Physics through Modeling of Electro magnetic fields phenomenon as lumped parameter circuit elements. {Leads to PO 1} In this case the CO-PO matching is very high. Search for questions related to this CO {SAR is also expected to point out such questions.} An example of such a question follows

A Question > R>0 C1 C2 Capacitors are of the same value, 1F; Initially, C1 is charged to V volts, and C2 is in a fully discharged state. The switch s closed at t = 0. Determine the energy dissipated in R at t = infinity

Let us analyze this question Does the question require just memory recall? NO! Can it be solved by direct application of text book material? NO! Does it require you to frame the problem in appropriate context? YES! Apply knowledge ( maths, science) to arrive at a solution. YES! Laws of conservation of energy and of electrical charges It is a closed system and conservation laws are applicable.

Implication The question is: Is the question appropriate for the CO1? CO1: {as per SAR}: Ability to apply laws of Physics through Modeling of Electro magnetic fields phenomenon as lumped parameter circuit elements. Yes, it is indeed. Next, we look for attainment.

Implication The question is: Is the question appropriate for the CO1? CO1: {as per SAR}: Ability to apply laws of Physics through Modeling of Electro magnetic fields phenomenon as lumped parameter circuit elements. Yes, it is indeed. Next, we look for attainment.

Assume that a question is Good, then We ask the question: Was the CO attained? For that, we look into answer scripts. We ask: How many of the students: (a) attempted?, (b) understood the question? (c) solved it correctly? (a), (b), and (c) together allow us to answer: How well was the CO attained. {Please record your observations and judgment}

Another Example Course is Electronic Circuits, CO: Apply Knowledge engineering specialization to the solution of complex engineering problems Judgment needed: (a) Is it a complex problem?{defined in slides29-30} (b) Does the solution require knowledge of engineering specialization? Here it knowledge of “engineering specialization”. COs can be many permutation combination. Not the entire PO may appear in a single CO.

A question as a Complex Engineering Problem Design an amplifier using BJT(s). Given: Signal source: 500mV (peak to peak), impedance 100K; Load:1K, Output required 1V (peak to peak) Problem does not state: (a) What amplifier configuration? (b) Reasoning behind the choice (c) What would be the typical device characteristics you visualize, etc For Tier -2 such a question could appear in a home assignment. For Tier-1,it could be design & build This is hard core engineering.

Analysis of the question The question does not state many aspects like: What amplifier configuration would be good for the given situation? Whether feedback should be applied? Answers to these questions require, “knowledge of engineering specialization (Electronics)” {PO 1} May be, also use of “Electronics Engineer’s Handbook” Attainment to be determined as given earlier Student may use electronics handbook. Reach different parts of a PO in different ways.

PO- Evaluation by Indirect Means Feedback from the Students After all they have undergone the Course

Analysis of the question The question does not state many aspects like: What amplifier configuration would be good for the given situation? Whether feedback should be applied? Answers to these questions require, “knowledge of engineering specialization (Electronics)” {PO 1} May be, also use of “Electronics Engineer’s Handbook” Attainment to be determined as given earlier Student may use electronics handbook. Reach different parts of a PO in different ways.

For Evaluators Could you set a CEP Exam question (If the time permits) in Your Domain? It will allow us to enrich our content, and the consequent discussion will bring in greater clarity.

PO- Evaluation by Indirect Means Feedback from the Students After all they have undergone the Course

Assessment Format Course EE 101 Course Title ___ Course Outcome - CO Attainment Level Scale 1 - 5 CO 1 – (Statement) CO 2 – (Statement) CO 3 – (Statement) CO 4 – (Statement) CO 5 – (Statement)

Format for Evaluation Mean η, Std. Dev. σ, Median μ Attain CO 1 2 3 4 5 CO 1 CO 2 CO 3 CO 4 CO 5 Mean η, Std. Dev. σ, Median μ Any suggestion for a metric? Please provide!

{To be verified by the Evaluators} Presumption Instructor has Explained the COs to the Students at the Start of the Course! {To be verified by the Evaluators}

Projects and Assignments Projects and Assignments can often lead to attainment of PO 1 and PO 2 and some other POs as well. Assessment: In Project Reports, Project Labs., and in the associated Marking Schemes. Evaluation will look for the following:

Evaluation of Projects Evaluation of Project Topics Method of Allotment (if any) Progress Seminars (during project execution)- Records of these. Evaluation of the state of the project in the final submission {This includes marking scheme used, level of completion, level of understanding in a study project, etc}

One needs take a holistic view in this evaluation. We look for evidence for all the POs (claimed in SAR) along with the bullets of the previous slide Typically among the POs could be (Other than PO1 to PO4): PO5, PO7, PO9, and PO11. Evidence for these POs could also be from Project assessment and Evaluation. One needs take a holistic view in this evaluation. This task can be a bit demanding Typically

Sample fourth year indicators for Problem analysis and Design* Identifies problem, known and unknown information, uncertainties, and biases Creates process for solving problem including justified approximations and assumptions Selects and applies appropriate model and analysis to solve problems Evaluates validity of results and model for error, uncertainty Identify problem and constraints including health and safety risks, applicable standards, economic, environmental, cultural and societal considerations *From Queens University report, Canada

Applies appropriate knowledge, judgement, and design tools, in creating and analyzing conceptual design solutions to select best concept Creates and tests simulations, models, and/or prototypes at various points in design with complexity appropriate to design stage Assesses design performance based on requirements, yield, reliability, and/or safety as appropriate Identifies possibilities for further improvement and conducts design review to evaluate performance of the overall process.

Laboratories & POs For some POs the Lab work is of Great help Need Lab records (Lab Journals) Instruction Manuals? Lab Viva Voce Structure of the Lab Experiments

Questions on Structure of Lab Work Are these Tasks rigidly Defined? How much time the students take to complete the Tasks? Excluding the time to write the journal. 15, 30, 60,120, minutes? Or Much more Any independent work they need to do? {Answers to these queries allows to Evaluate}

Typical ASSESSMENT Tool Types: Direct, Indirect

Assessment and Evaluation Qu. What is it that we are trying to assess? Ans. Have the Graduates qualified for the Profession. But, we assess the students continually as they progress through the program! So, most of the tools used here can be the ones that we use regularly, like: In-Semester exam, End-Semester exam, Tutorials, Quizzes, Assignments, and (may be) some more.

Qu. Then, what has changed? Ans. What has changed is: Now, we have to make assessments against the POs that we have declared as creating the required profile of the Graduate. Thus Assessment and Evaluation have to address this new requirement (and that is the Catch). These new needs influence the constructs of assessment and evaluation tools so that claims of COs and POs can be substantiated. {Critical for Tier-1}

Typical Assessment Tools Mid-Semester and End Semester Examinations** Tutorials* Home Assignments* Project work- Viva-Voce, Seminars etc. Employer/Alumni Feedback More

Attainment of Programme Outcomes Illustrate how course outcomes contribute to the POs Explain how modes of delivery of courses help in attainment How assessment tools, used to assess the impact of delivery of course/course content contribute towards the attainment of course outcomes/program outcomes 4. Extent to which the laboratory and project course work are contributing towards attainment of the POs { This is for the SAR. All the Evidence- that you put on the Table } Delivery of contents: Example- a blast furnace, Cut-out of a DG gear box. Or, in my course on electronic circuit design repeatedly small design tasks are given so that design experience builds up. It could also be a part of the lab.

Attainment of Programme Outcomes Illustrate how course outcomes contribute to the POs Explain how modes of delivery of courses help in attainment How assessment tools, used to assess the impact of delivery of course/course content contribute towards the attainment of course outcomes/program outcomes 4. Extent to which the laboratory and project course work are contributing towards attainment of the POs { This is for the SAR. All the Evidence- that you put on the Table } Delivery of contents: Example- a blast furnace, Cut-out of a DG gear box. Or, in my course on electronic circuit design repeatedly small design tasks are given so that design experience builds up. It could also be a part of the lab.

{ This is for the SAR. All the Evidence- that you put on the Table } Attainment of Programme Outcomes Programs are Expected to help you in this! Illustrate how course outcomes contribute to the POs Explain how modes of delivery of courses help in attainment How assessment tools, used to assess the impact of delivery of course/course content contribute towards the attainment of course outcomes/program outcomes 4. Extent to which the laboratory and project course work are contributing towards attainment of the POs { This is for the SAR. All the Evidence- that you put on the Table } Delivery of contents: Example- a blast furnace, Cut-out of a DG gear box. Or, in my course on electronic circuit design repeatedly small design tasks are given so that design experience builds up. It could also be a part of the lab.

Thanks Let Us Have a Break

Evaluation of the attainment of the Program Outcomes While writing the SAR Results of evaluation of each PO. (to be recorded) What are the levels of attainment? How the results of evaluation were used for curricular improvements? (Continuous Improvement -- Criterion-7)

SAR and the POs 2.2.2. Quality of end semester examination, internal semester question papers, assignments and evaluation (15) 2.2.3. Quality of student projects (20) 3.1. Establish the correlation between the courses and the Program Outcomes (POs) & Program Specific Outcomes (25) 3.2.2. Record the attainment of Course Outcomes of all courses with respect to set attainment levels (65) 3.3. Attainment of Program Outcomes and Program Specific Outcomes (75)

Purpose of Graduate Attributes Graduate attributes form a set of individually assessable outcomes that are the components indicative of the graduate's potential to acquire competence to practice at the appropriate level. The graduate attributes are intended to assist Signatories and Provisional Members to develop outcomes-based accreditation criteria While all attributes are important, individual attributes are not necessarily of equal weight.

Finally, SAR should have all the Evidence needed and should Show that all the program Outcomes are attained And That the Program meets all the Accreditation Criteria

One Question (?) In an Institution, all the inputs - Infrastructure, students, faculty, curriculum- are Good. Qu. Will the graduates be “NOT GOOD*?”? Ans. “May be- May be Not:” We just don’t know! In OBE, we determine {By Measurement} Twin question marks. In OBE we answer both the questions. What do we understand by “Good” and then determine How Good?

In OBE, We Define “Good”. and then How to determine “How Good”?

As Accreditation is all about Quality Assurance, We need to be Sure Which Implies OBE is a Good basis for it.

It appears to be very promising, But In the beginning, general feeling about OBE was: It appears to be very promising, But It needs is a more clear, and thoughtful exposition of what Outcome -Based Education really is, why is it needed, and how it operates

William Spady, (Father? OBE)

Qu. What does the term "Outcome-Based Education" really mean? “Outcome-Based Education: Critical Issues and Answers” by William G Spady Qu. What does the term "Outcome-Based Education" really mean?

Outcome-Based Education means clearly focusing and organizing every-thing in an educational system around what is essential for all students to be able to do** successfully at the end of their learning experiences. This means starting with a clear picture of what is important for students to he able to do, then organizing curriculum, instruction, and assessment to make sure this learning ultimately happens.

Roles of the two components Curriculum and the Teaching/ Learning processes are the basis on which the program is built Attainment of Outcomes indicates that the job is well done PC and PO are completely different in their roles.

Format for Evaluation Mean η, Std. Dev. σ, Median μ Attain CO 1 2 3 4 5 CO 1 CO 2 CO 3 CO 4 CO 5 Mean η, Std. Dev. σ, Median μ

Attainment of COs * indicates some elaboration is needed. 3.2 Attainment of COs 75 Evaluation Guidelines Observations??? 3.2.1 Describe assessment tools&processes 10 A. List Assessment processes(2) B. Quality /relevance of tools & processes(8) 3.2.2 Record of Attainment of COs 65 Verify Attainment Levels (65)* {How to determine Attainment Levels of COs?}* * indicates some elaboration is needed.

Criteria -3 Correlation Matrices Sr. No. Sub Criteria Max. Marks Evaluation Guidelines Marks Overall Marks& Grade Observations??? 3.1 A. Establish Correlation between CO-PO(5) B. COs embedded in Syllabi? (5) C. Explanation of Course articulation Matrix (5) D. Explanation Program articulation Matrix (10) 25 Evidence of COs for all courses (5)

Attainment of POs Observations??? 3.3 Attainment of POs 75 Evaluation Guidelines Observations??? 3.2.1 Describe assessment tools&processes 10 A. List Assessment processes(2) B. Quality /relevance of tools & processes(8) 3.2.2 Results of Evaluation of PO/PSO 65 A. Verification of documents, results and level of attainment of each PO/PSO (50)* B. Overall levels of Attainment(15) {Involves accumulation from all COs to all POs using the CO-PO matrix}

In Engineering, OBE Leads to: Defining of the Graduate Profile & It should be in line with the Washington Accord How do we do it?

NBA-Program Outcomes 1. Engineering Knowledge, 2. Problem Analysis 3. Design/development of solutions, 4. Conduct investigations of complex Problems, 5. Modern tool usage, 6. The engineer and society, 7. Environment and sustainability, 8. Ethics, 9. Individual and team work, 10. Communication, 11. Project management and finance, 12. Life-long learning Criteria 2 110