1. Methodological Issues in Needs Assessment for Quality Assurance in a National Context: The Case of Head Start Needs Assessment Hsin-Ling (Sonya) Hung, University of North Dakota James. W. Altschuld, The Ohio State University Contact: sonya.hung@email.und.edu Paper presentation at the Annual meeting of the American Evaluation association November, 2011 Anaheim, CA

2. Needs Assessment (NA) Basic Concepts  Needs are measured discrepancies between what is and what should be.  NA is “ the process of identify needs, prioritizing them, making needs-based decisions, allocating resources, and implementing actions in organizations to resolve problems underlying important needs” (Altschuld & Kumar, 2010, p. 20).

3. NA Basic Concepts  Scope of assessment (large, small) and general nature of needs area should be determined early in the process.  NA is usually conducted by organizations as related to organizational change, development, and use of resources. They can be political in nature.  NA should lead to implementation of an action plan to resolve needs.

4. Quality Assurance (QA) QA refers to:  Actions taken to ensure that standards and procedures are adhered to and that delivered products or services meet performance requirements.  Monitoring, evaluation or review in order to establish stakeholder confidence that products or services meets expectations or minimum requirements.

5. Needs Assessment & Quality Assurance  NA can be used for quality assurance  The purposes here are to identify gaps for improvement in performance or quality service and prioritizing for resource allocation  Head Start Needs Assessment is one case.

6. Head Start Needs Assessment  The Head Start Reauthorization Act (amended December 2007) requires each state’s Head Start Collaboration Office and offices for the American Indian, Alaska Native, Migrant and Seasonal Head Start Programs [collectively referred to here as Head Start Collaboration Offices, HSCO] to implement annually a statewide Head Start Needs Assessment (HSNA).

7. Head Start Needs Assessment  Initial one was in 2008-2009 with the intent of identifying needs in Head Start practices in 10 areas (Health Care, Services for Children Experiencing Homelessness, Welfare/Child Welfare, Child Care, Family Literacy Services, Services for Children with Disabilities, Community Services, Partnerships with Local Education Agencies, Transition and Alignment with K‐12, and Professional Development) regarding coordination, collaboration, and alignment of services.  Results inform the activities of the HSCO for strategic planning for improved services for children and families served by Head Start grantees.

8. The Head Start Needs Assessment National Analysis Project  Not initially planed to conduct a national analysis.  After compiling each state’s NA report, the Federal Office of Head Start found a need to understand a comprehensive picture of the national NA results and to identify the needs for technical assistance provision.  The University of Cincinnati Evaluation Service Center was selected to conduct the national analysis project.

9. Methodological Issues  Data Analysis was based on individual State’s NA reports and needs data obtained from each collaboration office through the Office of Head Start at the federal level.  Variations existed in sampling approaches, survey instruments, data collection procedures and coding schemes.

10. Sampling issues  No standard sampling procedure were established to guide each state’s NA activities.  But, the OHS emphasized the head start grant recipients as each state’s NA participants.  Each state decided their own sampling approach. Variations were in participant selection include Grantee-based, Grantee and delegate, and Program-based.

11. Participant Selection Approach FrequencyPercentage Grantee-based3266.7 Grantee and delegate 612.5 Program-based816.7 No Information Available2 4.2 N=48

12. Instrumentation issues  Omnibus instrument developed in D.C. may not exactly fit State and local conditions and agencies.  Not designed as typical NA instrument containing two conditions—what is and what should be—for rating on the same item.  Modification of language/wording of the instrument to fit local state’s need and language use caused consistency issue in instrument and data.

13. Modifications of Head Start Needs Assessment National Template Items Item Modification Categories Numbers of items modified in this way by one or more statesPercentage Wording change, same meaning15188.8 Item not included in the state instrument13579.4 Title or term change, same meaning4227.4 One item separated into multiple items3822.4 Multiple items combined into one item1911.2 One element retained but another excluded 105.9 Concept change52.9 Items with no modification or exclusion by a state 21.2 Number of items = 170

14. Data Issues  Use of existing data  Different data formats  Different scales employed  The data collection form worked and the respondents were providing information in accord with most of the questions and categories  There is a unit of analysis concern that may or not make a difference in analysis and subsequent interpretation

15. Type of Data Received Data file TypeFrequencyPercentage Excel file, raw data2961.7 Word file, aggregated data612.8 Excel file, aggregated data510.6 SPSS File48.5 Copies of original surveys24.3 Data embedded in report12.1 N=47

16. Data Collection Methods FrequencyPercentage Online survey/electronic mode 2756.3 Multiple data collection approaches 714.6 Mail survey24.2 No information available 1225.0 N=48

17. Lessons Learned  Preparation earlier in the process  NA trained personnel  Develop a standard sampling procedure for all States to implement state NA activities.  Develop a standard data collection protocol for all states to implement.

18. Lessons Learned  Use a consistent instrument across the nation. Revise the national template or select items for a core set of items fitting all states.  Allow state to develop additional sections to meet individual state’s concerns.  Consider using the NA concept of assessing the current and desired states in any future NA survey instrument.  Implement technical assistance based on priorities identified relative to the key issues recognized by the Office of Head Start.

19. Q & A  Questions related to the project  Questions or Suggestions for NA methodological concerns.  Other concerns or suggestions related to Need Assessment.

20. References  Altschuld, J. W. & Kumar, D. D. (2010). Needs Assessment: An Overview. Thousand Oaks, CA: Sage Publications.

21. Contact