1. 1 icfi.com | Social Security Administration Call Order 0001: Job Analysis Methodology Briefing of Final Project Results OIDAP Quarterly Meeting September 21, 2011 Lance Anderson, Ph.D. Brian Cronin, Ph.D.

2. 2 icfi.com | Introductions/Overview of Team –SSA –ICF –Subcontractors Introduction to Project and Purpose Project Methodology Recommendations –Job Analysis Procedures –Job Analysis Models Summary –Overarching Recommendations –Potential Next Steps Questions? Agenda

3. 3 icfi.com | Introductions

4. 4 icfi.com | SSA Debra Tidwell-Peters David Blitz Michael Dunn Elizabeth Kennedy Mark Trapani Overview of Team ICF International Brian Cronin, Ph.D. Lance Anderson, Ph.D. Beth Heinen, Ph.D. Jessica Jenkins, MPhil Allison Cook, M.S. Daniel Fien-Helfman Subcontractors Paul Davis, Ph.D. Kelly Day, OTD Len Matheson, Ph.D.

5. 5 icfi.com | Introduction to Project and Purpose

6. 6 icfi.com | SSA is developing new occupational information system (OIS) tailored specifically to SSA’s disability programs and adjudication process. –OIS will replace the Dictionary of Occupational Titles (DOT) and companion volumes, including the SCO and RHAJ. –To develop OIS, SSA needs detailed methodology and strategy for analysts to perform job analysis throughout U.S. labor market. –Purpose of Call 0001 was to perform the research needed to identify useful features of existing practices to support development of SSA’s job analysis methodology. Introduction to Call Order 0001

7. 7 icfi.com | Project Methodology

8. 8 icfi.com | Subtask 3.1: Develop Results Template Subtask 3.2: Evaluation of Major Job Analysis Practices Task 1: Task Meetings/Discussion Task 2: Review and Evaluation of Job Analysis Practices Task 3: Report Job Analysis Practice Results Project Methodology Subtask 2.1: Documentation of a Detailed Project Methodology Subtask 2.2: Literature Review/Background Research Subtask 2.3: Documentation for SSA

9. 9 icfi.com | Overarching Goal: Summarize wealth of information on job analytic procedures across disciplines to allow SSA decision makers to quickly understand and evaluate various job analysis practices. To ensure consistency, we use the following nomenclature: –Project Method(ology) – Refers to steps taken to conduct research for this call order –Job Analysis Method(ology) – Refers to ultimate SSA job data collection process developed to address OIS needs –Practice – Refers to all job analysis approaches, models and procedures, identified through this call order –Model – Refers to an established, ‘off-the-shelf’ job analysis approach identified through this call order –Procedure – Refers to a data collection technique identified through this call order Call Order 0001 Terminology

10. 10 icfi.com | Recommendations on Job Analysis Procedures

11. 11 icfi.com | Recommendation: Effective and cost-efficient starting point for almost all job analysis. Should be included in SSA’s ultimate data collection methodology. Potential Usage for SSA: Analysts are able to become more familiar with the target job. Potential Challenges: Dependent upon the usefulness and availability of source documents. Should only be used in conjunction with other data collection procedures. Outcome: Knowledge gained can be used to inform development of subsequent data collection techniques (e.g., interviewing). Procedure: Review of Written Materials

12. 12 icfi.com | Recommendation: Use for jobs that include more manual, less cognitive tasks. Should be included in SSA’s final data collection methodology. Potential Usage for SSA: Useful for collecting detailed information about job tasks, equipment/materials used, and work environment; does not rely solely on testimony of incumbents. Potential Challenges: Can be costly and time-consuming; may not be appropriate for all jobs (e.g., highly cognitive jobs). Outcome: Should be used in combination with other data collection procedures to determine prevalence of work activities observed. Procedure: Job Observation

13. 13 icfi.com | Recommendation: Use when gathering data from geographically dispersed incumbents. Should be further considered for inclusion in SSA’s ultimate methodology. Potential Usage for SSA: Can be effective and efficient means of collecting data from numerous incumbents; useful for assessing prevalence of work activities/; can make comparisons across jobs. Potential Challenges: May be costly to develop and administer; Some threats to validity, such as inflated ratings, lack of understanding rating elements, lack of effort in completing survey. Outcome: Other data collection procedures should be used to supplement the data collected especially when more detail is needed. Procedure: Survey

14. 14 icfi.com | Recommendation: Use when job is more complex and needs additional clarification or when comparison is needed (as long as highly structured). Should be incorporated into SSA’s ultimate data collection methodology. Potential Usage for SSA: Allow analysts to collect detailed job information through the direct questioning of incumbents. Potential Challenges: Can be time-consuming and costly to prepare protocol, coordinate participants’ schedules, and arrange travel (for face-to-face interviews). Outcome: Analyst should undergo training on interviewing skills; analysts should conduct multiple interviews to gain full value of this procedure. Procedure: Structured Interviews

15. 15 icfi.com | Recommendation: May be more time-efficient and cost-efficient than interviews, but we recommend that interviews be used instead of focus groups whenever possible to allow for clarification questioning. Potential Usage for SSA: Allows for discussion with multiple incumbents at one time; useful for idea generation or other data collection that depends on interaction to elicit information. Potential Challenges: Group dynamics can influence participant responses; schedule coordination can be difficult; specialized training needed for job analysts to be skilled facilitators. Outcome: Participants should be provided with an agenda, ground rules, and background information prior to the session. Procedure: Focus Groups

16. 16 icfi.com | Recommendation: May be valuable for jobs requiring manual labor given the disability determinations SSA must make. Depending on SSA’s ultimate construct model and data collection instrument, should include instrument measurement of physical demands (i.e., tools that measure PDs) but with limited use. Potential Usage for SSA: Advantages include precise nature of the collected data, high reliability, high validity, and data that are typically easy to aggregate. Potential Challenges: Can be resource intensive to administer and require specialized training; can be intrusive to incumbents. Outcome: Must have technical expertise in the use of specific measurement tools selected. Procedure: Physical Demands Measures

17. 17 icfi.com | Recommendations on Job Analysis Models

18. 18 icfi.com | Model: AET Examples of Effective Features Use of descriptors that isolate specific types of physical effort Use of scales that focus on frequency, duration, and significance Use of examples to assist in coding level of demand Example Limitations for SSA Context Significant time and labor commitment to develop career ladders Long term commitment to upkeep of career ladders The AET involves conducting an observation and interview to complete an ergonomic questionnaire.

19. 19 icfi.com | Model: Common Metric Questionnaire (CMQ) Examples of Effective Features Matrix structure of the questionnaire Computerized interface to allow for effective use of the questionnaire Use of behavioral and observable descriptors that are easy for incumbents and supervisors to rate Example Limitations for SSA Context Does not include a comprehensive set of descriptors at a broad level Some respondents might not have the access to a computer to use the computerized interface CMQ-like items on the OIS would need to be continually updated The CMQ collects data via a survey administered directly to incumbents and/or their immediate supervisors.

20. 20 icfi.com | Model: Cognitive Task Analysis (CTA) Examples of Effective Features Use of structured observation and interviews together with completion of a structured tool by the analyst Identification of the various types of knowledge needed to do the job Example Limitations for SSA Context Tends to lack the detailed information needed by SSA on various physical abilities Would require extensive training of analysts The CTA approach involves using a variety of data collection procedures to ultimately identify the cognitive processes underlying a job with a particular focus on the processes that distinguish an expert from a novice.

21. 21 icfi.com | Model: Fleishman Job Ability Requirement Scales (F-JAS) Examples of Effective Features Generally well supported by research Use of level scales anchored with observable behaviors Example Limitations for SSA Context Lacks some generalizable physical abilities constructs that are important to SSA Tends to provide details on the variation of jobs at the high end of many abilities, whereas SSA might be more interested in information about the variation in the jobs at the low end of abilities The primary data collection procedure for the F-JAS model involves the administration of the Ability Requirements Scales to collect data on 52 types of abilities.

22. 22 icfi.com | Model: Functional Job Analysis (FJA) Examples of Effective Features Inclusion of work context and worker environment variables Use of procedures that can be easily trained Structured framework and structured protocols build validity Example Limitations for SSA Context Lacks standardization on important issues such as how jobs are sampled, how interviews are conducted, and how many interviews are conducted The DOT scales lack detail on cognitive abilities and interpersonal skills The FJA approach gathers a variety of different types of job analysis data typically collected via interview and observation but may also include other data collection procedures.

23. 23 icfi.com | Model: Job Element Model (JEM) Examples of Effective Features Approach to conducting interviews and observations to gather job specific information provides data for understanding the job Example Limitations for SSA Context Experts previously rated this model low in terms of reliability and standardization Although a low cost approach, it involves a significant amount of time to administer JEM focuses on the human attributes required for superior performance on the job and collects data via focus groups, interviews, and surveys.

24. 24 icfi.com | Model: O*NET Examples of Effective Features The hierarchical arrangement and use of the content domain A nationwide database supported and maintained by an external entity Example Limitations for SSA Context Some items will likely have low reliability relative to other instruments because constructs are not observable Tends to focus on differentiating high-tech jobs as opposed to low skill jobs that are usually the focus of disability claims O*NET was developed using a job analysis approach that focuses primarily on surveys, with supplementary use of interviews and reviews of written material.

25. 25 icfi.com | Model: Position Analysis Questionnaire (PAQ) Examples of Effective Features Use of an instrument that focuses on generalizable work activities provides data for cross-job comparison Focus on observable behaviors ensures greater verifiability of the findings Example Limitations for SSA Context Constructs at a level of abstraction that may not provide a clear picture of the job. The PAQ is an existing job analysis model that uses a standardized 195- item instrument to collect data, which is typically completed by a job analyst based upon data collected in job observations and interviews.

26. 26 icfi.com | Model: Task Inventory Examples of Effective Features Method of conducting interviews and observations to gather job specific information provides data for understanding the job Example Limitations for SSA Context No generalizable constructs or scales to allow for cross-job comparison The Task Inventory approach involves collecting data through procedures such as review of written materials, job observation, interviews, and surveys to ultimately develop a list of task descriptions.

27. 27 icfi.com | Model: Threshold Traits Analysis (TTA) Examples of Effective Features The 33 traits include a parsimonious and simply worded set of constructs that might provide an effective perspective for sorting and locating jobs In general, the TTA is well supported by research Example Limitations for SSA Context SSA would need more detail than is provided via the 33 trait focused scales In our judgment, the TTA rating tool is not appropriate for use as incumbent/supervisor survey. It should only be used by trained analysts The TTA collects worker trait, job demand, and job function data using data collection procedures such as review of written materials, job observations, interviews, and surveys.

28. 28 icfi.com | Summary: Overarching Recommendations and Potential Next Steps

29. 29 icfi.com | Identify Work Taxonomy and Constructs to be Measured –Measuring different constructs may necessitate different data collection procedures Data Should be Collected and Stored Using a Computerized System or Online Application/Tool –Provides a centralized location for data collection and minimize potential security issues/concerns Need to Determine Factors that are Most Important and Consider Job Analysis Practices Accordingly Fully conceptualize multiple prototypes of integrated systems that SSA might use and compare the systems side-by-side –Integrated system= Occupational analysis system Recommendations and Potential Next Steps

30. 30 icfi.com | Comparing Prototype Integrated Systems (i.e., Occupational Analysis Systems) Side-by-Side DesignPrototypeEvaluate

31. 31 icfi.com | Full Methodology Must Include a Comprehensive Set of Procedures that Include Guidelines for Maintaining Data Security and Confidentiality Features of Existing Job Analysis Models Should be Adapted for SSA’s Specific Purposes and Data Needs –Additional job analysis questions or more precise questions may need to be added to an existing questionnaire Need Pilot Testing to Ensure that the Final Methodology Meets SSA Objectives –Ensures the final set of combined data collection procedures and/or models appropriately measure the desired characteristics Recommendations and Potential Next Steps

32. 32 icfi.com | Questions? For more information, please contact: Dr. Brian Cronin Senior Manager ICF International (512) 388-3389 BCronin@icfi.com Dr. Lance Anderson Vice President ICF International (703) 934-3000 LAnderson@icfi.com