1. A Framework for Assessing Needs Across Multiple States, Stakeholders, and Topic Areas Stephanie Wilkerson & Mary Styers REL Appalachia American Evaluation Association Conference, October 16, 2013

2. Presentation Overview Provide background on REL Appalachia Present needs assessment framework Provide examples of needs sensing activities Describe challenges and solutions for each step in the needs assessment process

3. Background: REL Appalachia One of ten Regional Educational Laboratories funded by the Institute of Education Sciences 5-year contract; currently in Year 2 Serving Kentucky, Tennessee, Virginia and West Virginia Providing analytic technical assistance, research, and information dissemination to build capacity for data use Focusing on college and career readiness, supporting effective teachers and leaders, and low achieving schools

4. What drives REL Appalachia’s work? Needs of stakeholders across the region Who are our stakeholders? – Governing Board – State departments of education – Local educators – State association and education officials – Research alliance members

5. REL Appalachia Needs Assessment Reporting Bi-Annual Communities Served Reports – Due February and August of each year Annual Assessment of Analytic Needs – Due November of each year Updated Annual Plan – Draft due October of each year (start planning in June)

6. REGIONAL NEEDS EducatorsAsk A REL Regional news, publications, policy and legislation reviews State association and education officials Research Alliance members Governing Board members What are our sources for identifying needs?

7. 1. IDENTIFY AUDIENCES 6. GUIDE SERVICE AGENDA 5. ANALYZE NEEDS 3. DESCRIBE NATURE AND EXTENT OF NEEDS 4. CHARACTERIZE SERVICE COMMUNITIES 2. DEFINE THE NEED How do we assess needs? Six-Step Needs Assessment Process

8. Step 1: Identify the Audience This step addresses the following questions: – Who will use this information and for what purposes? – Do intended users have information priorities that might influence the data collection schedule? What we found: – Primary stakeholders are REL Appalachia staff – Timeframe for needs data revolves around annual planning

9. Step 1: Challenges & Solutions Challenge: – Timeframe for using needs data does not align with deadline for needs assessment report Solutions: – Create informal feedback loops to convey current needs data (e.g., internal summary memos) – Contribute to planning discussions – Build information databases that capture needs and is easily accessible to REL Appalachia staff

10. Step 2: Define the Need This step addresses the following questions: – What need is identified? – Is the need clearly defined? – What do stakeholders perceive as possible solutions for addressing this need? What we did: – Conducted fishbone activity with Governing Board – Added these questions to REL Appalachia’s internal database – Used multiple data sources

11. Step 2: Challenges & Solutions Challenges: – Survey development and administration not feasible within contract year – Limited to collecting data from a sample of nine or fewer for each stakeholder group Solutions: – Changed data collection method from survey to phone interviews – Developed 16 tailored interview protocols specific to key issues within each state

12. Step 3: Describe Nature & Extent of Needs This step addresses the following questions: – How widespread is the need? – What do we already know about the need and possible solutions? – What additional information do we need? What we did: – Conducted literature scans – Consulted multiple data sources – Considered work conducted by other RELs

13. Step 3: Challenges & Solutions Challenge: – Hard to determine how widespread a need is with limitations on sample size for data collection methods Solutions: – Used informal communication networks among REL staff for sensing needs as they interact with stakeholders in the field

14. This step addresses the following questions: – What are the characteristics of the communities served by REL Appalachia? – What is the focus, nature, and intensity of services they are receiving? – Are there gaps in service coverage? What we did: – Reviewed internal documentation on research alliances – Considered how other service providers might be addressing needs – Examined customer satisfaction data related to needs Step 4: Characterize Service Communities

15. Step 4: Challenges & Solutions Challenge: – Describing nature of engagement with alliances – Describing intensity of services provided to alliances – Representing distribution of services; identifying gaps Solutions: – Having REL Appalachia alliance researchers characterize their engagement with alliances according to Himmelman’s collaboration framework – Documenting and quantifying delivery of services in information database – Using GIS mapping to give visual representation of service coverage across region

16. Step 4: Describing Communities Served ConstructDescriptionData Source Research alliance composition Number and type of members; geographic distribution REL AP internal documentation Alliance focusAlliance goals; current and planned work Year 2 Updated Annual Plan; monthly progress reports Nature of engagementBased on Himmelmann’s framework for collaborating, cooperating, and networking REL AP Services and Needs database Service intensityNumber of contracts made and services delivered; number of participants reached; service purpose REL AP Services and Needs database Ensuring service coverageREL AP contacts with non-alliance members; regional needs sensing activities Ask A REL; needs assessment; news and literature scans; Services and Needs database

17. Step 5: Analyze Needs This step addresses the following questions: – Which needs are most prevalent, relevant, and important? – How can we feasibly and effectively address these needs? What we did: – Consulted multiple and varied data sources – Collected information representing regional, state, and local needs – Contributed to planning discussion about feasibility of responding to expressed needs

18. Step 5: Challenges & Solutions Challenges: – Engaging internal stakeholders in dialogue about findings – Synthesizing various types of data from multiple sources – Determining which needs should be prioritized Solutions: – Became more timely in presenting needs data – Developed a three-step synthesis process 1.Analyzed data from each source separately and moved key findings into synthesis database 2.Applied prioritization criteria to key findings for each source 3.Conducted content analyses across prioritized, key needs

19. Step 5: Criteria for Prioritizing Needs

20. Step 6: Guide Service Agenda This step addresses the following questions: – How are we responding to regional needs? – How can we communicate our response to our stakeholders? – Are new needs surfacing?

21. MEETING STAKEHOLDER NEEDS Bridge events Ask A REL responses Research briefs & newsletters Research studies Workshops, roundtables & webinars Governing Board meetings Step 6: Responding to Regional Needs

22. Step 6: Challenges & Solutions Challenge: – Communicating REL Appalachia’s response to needs assessment results Solutions: – Present agenda to Governing Board members – Highlight work on website and in newsletters

23. Lessons Learned Needs assessment team members need to be integrated into the alliance work – To conduct informal needs sensing – To identify cross-alliance needs – To make more informed recommendations for how needs data relate to current and future work Data collection activities need to occur according to planning deadlines, not deliverable deadlines Data collection must balance identifying broad regional needs with targeted alliance needs Needs data must be communicated in a timely manner with attention to relevance and feasibility Building internal capacity to informally assess and document needs is critical

24. Questions? Contact – Stephanie Baird Wilkerson  stephanie@magnoliaconsulting.org stephanie@magnoliaconsulting.org – Mary Styers  mary@magnoliaconsulting.org mary@magnoliaconsulting.org Visit – www.relappalachia.org www.relappalachia.org – www.magnoliaconsulting.org www.magnoliaconsulting.org