1. Establishing a Culture of Assessment Evaluation of the Effectiveness
Alabama A&M University
January 07, 2008
John D. Jones, Ph.D.
Associate Provost for Institutional Effectiveness and Research
Bennett College

2. Goals for the Workshop
clear understanding of how to develop an assessment plan
more knowledgeable about assessment methods
more knowledgeable about analyzing and reporting assessment results
more knowledgeable about how to assure the quality of assessment plans and the reporting of assessment results

3. What Do We Mean by Assessment?
it is a formative evaluation process designed to support improvement
it is continuous
it is focused on improvement
student learning
student development
the institution and its people

4. Levels of Assessment classroom assessment course assessment
assessment of individual students performance at the course level by an instructors
course assessment
assessment of how well a course is meeting student learning outcomes
program assessment
assessment of how well an academic program is meeting student learning outcomes
assessment of how well a support program is meeting its objectives
institutional assessment
assessment of campus-wide issues or programs

5. Institutional Effectiveness and Program Assessment
focuses on continuous quality improvement
academic programs
administrative support units
examines students learning outcomes, program outcomes, customer satisfaction, and unit performance to identify areas to improve
leads to actions
conducted annually

6. Purposes of Program Assessment
to improve
the assessment process should provide feedback to determine how the program can be improved
to inform
the assessment process should inform faculty and other decision makers of the contribution and impact of the program
to prove
the assessment process should encapsulate and demonstrate what the program is accomplishing to students, faculty, staff, and outsiders
to support
the assessment process should provide support for campus decision-making activities such as program review, strategic planning, and external accountability activities such as accreditation

7. Effective Program Assessment Should Answer these Questions
What are you trying to accomplish?
How well are you doing it?
How, using the answers to the first two questions, can you improve what you are doing?
What and how does a program contribute to the development and growth of its students and/or the support of its customers?
How can student learning be improved?

8. Program Assessment is Effective When Assessment…
is viewed as comprehensive, systematic, and continuous activity
is viewed as a means for self-improvement
measures are meaningful
utilizes multiple measures and multiple sources
is used as a management tool
results are valued, and are genuinely used to improve programs and processes
has sufficient coordination and review
includes all constituents in the process (e.g., faculty, staff, administrators, students, community, industry)

9. Mechanics of Assessment
assessment is a continuous improvement process
to improve, you need to know where you are today and where you would like to go
mission (purpose)
vision (where you would like to go)
goals (steps to getting where you would like to be)
objectives or outcomes (what you need to achieve in order to get there)
measures (how well you are currently doing)
to improve, you need to take action
analyze your program or operations to determine changes
plan the changes
take action

10. Organizing for Assessment
before assessment can begin, key players, committees and structures must be identified and assume responsibility for designing, implementing, and carrying out the assessment process
understanding the needs of program or unit can you help think about the design of the assessment plan
depending on purpose, the plan can be informal (for internal use) or formal (external audience)

11. Identify the Scope of the Plan
What should the assessment include?
assess resources (facilities, students, faculty)
assess processes (pedagogy, advising, feedback)
assess results or outcomes
who/what gives you feedback
academic program assessment typically focuses on student outcomes
administrative unit assessment typically focuses on quality of products, processes, and services

12. Defining the Mission Statement
a broad statement of what the program or unit is, what it does, and for whom it does it
a clear description of the purpose of the program or unit and the learning environment
reflects how the program contributes to the education and careers of students graduating from the program or how the unit supports its customers
aligned with department, college, and university missions
distinctive for your program or unit

13. Components of a Mission Statement
primary functions or activities of the program or unit
most important functions, operations, outcomes, and/or offerings of your program or unit
purpose of the program or unit
the primary reasons why you perform your major activities or operations
stakeholders
groups or individuals that participate in the program and those that will benefit from the program or unit

14. Structure of a Mission Statement
“The mission of (name of your program or unit) is to (your primary purpose) by providing (your primary functions or activities) to (your stakeholders).” (Additional clarifying statements)
(Note: the order of the pieces of the mission statement may vary from the above structure.)

15. Checklist for Mission Statement
Is the statement brief and memorable?
Is it distinctive?
Does it clearly state the purpose of the program or unit?
Does it indicate the primary function or activities of the program or unit?
Does it indicate who the stakeholders are?
Does it clearly support the department’s, college’s, and university’s missions?

16. Defining Program Goals
establishment of goals prior to developing objectives is recommended
some assessment processes don’t include reporting of goals
goals are long-term organizational or program targets or directions of development
they state in broad terms what the organization wants to accomplish or become over the next several years
goals provide the basis for decisions about the nature, scope, and relative priorities of projects and activities
goals should help move the organization or program to attain its vision

17. General Process Used in Generating Goals
examine mission
think what that unit or program would look like and how its services and operations (refer to you mission) would need to be conducted to reach that vision
improve, minimize, maximize, provide
generate list of potential goals and prioritize
state these as goals
two general approaches to developing goals
review existing documents
ideal unit or program approach

18. Defining Objectives and Student Learning Outcomes
a measurable target with a time limit that must be met on the way to attaining a goal
student learning outcome
specific statements that describe the intended learning outcomes that must be met on the way to attaining the degree
typically derived from goal statements
more precise, more specific and measurable than the goal statement
there can be more than one objective or outcome related to each goal

19. Think SMART When Defining Student Outcomes and Program Objectives
Specific
clear and definite terms describing the abilities, knowledge, values, and attitudes
Measurable
it is feasible to get the data, data are accurate and reliable, it can be assessed in more than one way
Aggressive but Attainable
don’t let the perfect divert you from what is possible
Results-oriented and Time-bound
describe where you would like to be within a specified time period
describe what standards are expected from students or what percent of students are expected to achieve a particular level of performance

20. Agenda Identifying Student Learning Outcomes
What is a student learning outcome
Writing student learning outcomes
Types of student learning outcomes
Identifying key outcomes

21. Agenda…continued Measuring Student Learning Outcomes Direct evidence
Indirect evidence

22. Student Learning Outcomes
What is a Learning Outcome?
Knowledge, skills and attitudes that students take with them from a learning experience.

23. Writing Learning Goals
Aim for goals that are neither too broad or too specific
Use concrete action words
Define fuzzy terms
Focus on the end, not the means
Focus on the most important goals
Work collaboratively with colleagues

24. Types of Learning Goals
Increase knowledge and basic understanding
Memorization
Replicating a simple procedure
Defining concepts
Summarizing concepts
Explaining concepts

25. Types of Learning Goals
Thinking, Performance, and Interpersonal Skills
Analysis, evaluation, problem solving, and decision making
Physical skills related to manipulating ideas, concepts, tools etc.
Listening, working with people from diverse backgrounds, teamwork, leadership ability

26. Types of Learning Goals
Attitudes and Values
Appreciation
Awareness of one’s own values, attitudes, and opinions
Integrity
Character
Valuing learning

27. Most Valued Learning goals Today!
Communication skills
Information literacy and research skills
Thinking skills
Interpersonal skills

28. Identifying key learning goals
Identify resources for potential goals
Make it a collaborative process
Achieve consensus on common goals
Preserve academic freedom
Share goals with students

29. Measuring Student Learning Outcomes
Direct evidence of student learning is tangible, visible and self-explanatory evidence of exactly what students have or have not learned.

30. Examples of Direct Evidence
Ratings of student skills by their field experience supervisors.
Scores and pass rates on appropriate licensure/certification exams.
Capstone experiences such as research projects, theses, dissertations, exhibitions, performances scored using a rubric.
Other written work or performance, scored using a rubric.
Portfolios of student work.

31. Examples of Direct Evidence… continued
Scores on locally-designed tests such as final examinations in key courses, qualifying exams, and comprehensive exams, accompanied by test blueprints describing what the test assess.
Score gains between entry and exit on published test, local test, or writing samples.
Employer ratings of the skills of recent graduates.
Student reflections on their values, attitudes, and beliefs.

32. Measuring Student Learning Outcomes
Indirect evidence of student learning provides signs that students are probably learning , but the evidence of exactly what they are learning is less clear and less convincing.

33. Examples of Indirect Evidence
Course Grades
Assignment grades, if not accompanied by a rubric or scoring guide
Admission rates into graduate school or graduation rates
Placement rates of graduates into appropriate careers and starting salaries
Alumni perceptions of satisfaction and career responsibilities
Student ratings of knowledge and skills
Honors, awards, and scholarships earned by students and alumni
Student/alumni satisfaction with their learning

34. More Details on Student Learning Outcomes
describe specific behaviors that a student of your program should demonstrate after completing the program
focus on the intended abilities, knowledge, values, and attitudes of the student after completion of the program
What is expected from a graduate of the program?
What is expected as the student progresses through the program?

35. Why Are Student Learning Outcomes So Important?
basis for program improvement
instruction, course design, curricular design
communicate instructional intent
included in the syllabus
increase awareness of learning (for students)
common language
advising materials
promotional materials
support accreditation and evaluation

36. When Defining Student Learning Outcomes
frame the learning outcome in terms of the program rather than individual courses or individual students
each student will receive a B or better in the XYZ course
graduates from the program will demonstrate knowledge of engineering fundamentals
use concrete action verbs (e.g., define, classify, operate, formulate) rather than passive verbs (e.g., be exposed to) or vague verbs (e.g., understand, know)
action verb lists based on Bloom’s Taxonomy

37. Types and Levels of Student Learning Outcomes (Bloom’s Taxonomy)
cognitive: recall and intellectual skills
knowledge, comprehension, application, analysis, synthesis, evaluation
affective: attitudes, values, interests, appreciation and feelings toward people, ideas, places, and objects
receiving, responding, valuing, organization, characterization by, value
skills
- perception, set, guided response, mechanism, complex overt response, adaptation, origination

38. When Defining Student Learning Outcomes (SLO)
do not to join elements in one statement that can not be assessed by single method
BSHE graduates will demonstrate knowledge of math, science, and engineering fundamentals, and gain competency in conducting oral presentations.
SLO statements should focus on the learning results and not on the learning process
computer applications will be introduced in all core engineering courses
BSHE graduates will demonstrate proficiency in XXX computer applications

39. When Defining Student Learning Outcomes (SLO)
SLO statements should be stated so that the outcome can be measured by more than one assessment method
students completing the XYZ engineering program will score over 95% on a locally-developed exam that tests application of engineering principles
students completing the XYZ engineering program will demonstrate competence and the ability to apply engineering principles
SLO statements should indicate the level and type of competence that is required of graduates of a program

40. Checklist for Student Learning Outcomes
aligned to mission and goal statements
clearly describe and define expected abilities, knowledge, values, and attitudes of the graduates of the program
simply stated
distinctive and specific to program
stated so that a single method can be used to measure the outcome
stated so that more than one measurement method can be used
focus on the learning results and not the learning process
measurable and there are resources available
can be used to identify areas to improve

41. Describing Measurement Methods
What are you going to use?
presentation, assignment, test, survey, observation, performance rating
Of and/or by whom?
student, mentor, focus group, customer, process, course
Context (e.g., where or when)
point-of-service, capstone, throughout the year, end of program
For what purpose
desired objective or learning outcome
Example: Test the students at the end of the program for their level of knowledge in XYZ

42. Curriculum or Course-based
performance-based
capstone courses
capstone projects
case studies
classroom assessment
course-embedded assignments
course-embedded exam questions
portfolios
other
curriculum and syllabus analysis
content analysis of courses
reflective essays

43. After Identifying the Potential List of Measures You Need to…
select the “best” ones
try to identify at least two measure per objective or outcome
identify performance targets
balance between stretch targets versus achievable targets
example
survey (using the Graduating Senior Survey) the students at the end of the program for their level of satisfaction with their communication skills (indirect method)
90% or more of the students will rate their level of satisfaction with their communication skills as “very good” to “excellent” on a survey (using the Graduating Senior Survey) that the students complete at the end of the program

44. Selecting the “Best” Assessment Methods
relationship to assessment—provide you with the information you need
reliability—yields consistent responses over time
validity—appropriate for what you want to measure
timeliness and cost—preparation, response, and analysis time; opportunity and tangible costs
motivation—provides value to student or customer, respondents are motivated to participate
other
results easy to understand and interpret
changes in results can be attributed to changes in the program

45. Challenges and Pitfalls
one size does not fit all—some methods work well for one program but not others
don’t try to do the perfect assessment all at once—take a continuous improvement approach
allow for ongoing feedback
match the assessment method to the objective and not vice-versa

46. After Identifying the Potential List of Measures You Need to…
develop assessment instruments
surveys
exams
assignments
scoring rubrics
portfolios
ideally you want them to be reliable, valid, and cheap
approaches
use external sources
seek help from internal sources (e.g., Institutional Research Office)
do it your self
the instrument may need to be modified based on assessment results