1. TAH Project Evaluation Data Collection Sun Associates

2.  Jeff Sun  jsun@sun-associates.com  www.sun-associates.com/tah  We’ve evaluated 11 TAH projects over the past 4 years  Program evaluators for TAH and non-TAH projects  Instructional Technology  STEM  Creating evaluation plans as part of strategic planning initiatives

3. Data in Isolation is No Data at All  Data collection must be rooted in performance indicators that are in-turn aligned with project goals  What are we going to do?  How do we know when we’ve done it/are doing it?  What data supports this conclusion?  Data needs are defined by the indicators  An example…

4. Question 1  Our first question is largely quantitative

5. Question 2  It turns out that Question 2 is much more qualitative  But this in fact address more of the project’s goals

6.  Data collection is linked to indicators  Projects need quantitative as well as qualitative data to respond to their indicators  Quantitative data is pretty straight-forward  Tests (teacher and student)  Counts (enrollments, participation, etc.)  Qualitative data requires more work and more specialized skills to collect and analyze

7. Qualitative Methods  Rubrics and Checklists  Essentially like “grading criteria”  Lists/examples of behaviors that constitute acceptable performance  Historical Thinking Skill benchmarks  Other “effective teaching” rubrics – created by project staff  Rubrics for lesson plans/participant work products  Bottom line is that work and/or performance is assessed against these rubrics…and the results are data that go toward assessment of the project indicators

8.  Classroom observations  Basic data collection – “What Actually Happened?”  Teacher de-brief/interview  Recording observations later on a indicator-aligned template  Meeting observations  We consider staff meetings, board meetings, planning meetings, etc. as legitimate data collection events  As evaluators, we need to observe everything that goes into operating this project

9.  Focus groups  Engaging participants in a focused discussion around questions that are aligned with the project indicators  Seeking normative statements as well as evidence of a disparity of opinion  But we’re looking for opinions  Occur several times throughout the project year…at minimum at the beginning and end of each participant cohort  Selection issues (who selects the participants) need to be considered  Can also be conceptualized as 1-1 interviews

10.  Surveys  The basic “workshop evaluation form”  Given our typical sample size (20 or so participants per session) we look for open-ended responses versus numerically scaled responses.  Online surveys allow for easy use of text responses  We also use surveys for background needs assessment – which can be used to substantiate later claims that the project has “met participants’ needs”.

11.  Control Groups  Not all of our evaluations make use of control group methodology  This is not appropriate – or practical – for many project evaluations (and we would argue the former is true for most TAH projects)  When necessary, we have operated control groups for quantitative data (tests of teacher/student knowledge)  Control groups can also generate qualitative data, but extra work and expense is of little practical benefit in most TAH projects

12. Pulling it All Together  Quantitative data provides quick facts that can be input into the ED 524B form …  But these “facts” can be easily misinterpreted or misleading  Quantitative data alone doesn’t provide any context for understanding what the numbers may mean  Qualitative data provides context and deep description of project actions and outcomes  Qualitative data provides considerable text for written evaluation reports…and reflective, formative, discussion of project performance in meeting its indicators  Which we see as the primary value of project evaluation

13.  Jeff Sun  jsun@sun-associates.com  www.sun-associates.com/tah