Stages of Development (SoD) National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR) Copyright National Institute on Disability, Independent Living, and Rehabilitation Research 9/16/2015 1

Purpose These PowerPoint slides provide additional information on NIDILRR’s proposed Stages of Development (SoD). These slides should be read in conjunction with the Stages of Development (SoD) Overview. Stages of Development (SoD) Overview Questions pertaining to use of these slides can be ed to SoDComments.SoDComments We will respond as quickly as possible! Copyright National Institute on Disability, Independent Living, and Rehabilitation Research 9/16/2015 2

Introduction NIDILRR’s Mission “Generate new knowledge and promote its effective use in improving the ability of persons with disabilities to perform activities of their choice in the community, [And] To expand society's capacity to provide full opportunities and accommodations for its citizens with disabilities.” Copyright National Institute on Disability, Independent Living, and Rehabilitation Research 9/16/2015 3

Introduction NIDILRR achieves this mission by funding grants comprising research, development, training, technical assistance, capacity building, and knowledge translation activities. Research and development activities receive the largest portions of these investments. NIDILRR seeks to optimize benefits to Persons with Disability derived from these investments. Copyright National Institute on Disability, Independent Living, and Rehabilitation Research 9/16/2015 4

Introduction Previously, NIDILRR developed and integrated the Stages of Research (SoR) into its Long Range Plan ( ).Long Range Plan ( ) The SoR have been used to structure and improve: solicitations, expert recruitment for peer-review panels, peer-review protocols, annual reports, grant monitoring, performance metrics, strategic planning, and dissemination of findings. NIDILRR anticipates similar uses for the proposed Stages of Development (SoD). Copyright National Institute on Disability, Independent Living, and Rehabilitation Research 9/16/2015 5

Concepts Research – comprises activities to create new knowledge using appropriate scientific methods. Research findings are often documented in peer-reviewed articles and disseminated through journal publications. Development – comprises activities to create new or improved products using appropriate development methods. Development products are often adopted by targeted population(s) and used for intended purpose(s). Copyright National Institute on Disability, Independent Living, and Rehabilitation Research 9/16/2015 6

SoD Overview The proposed SoD has three stages: the Proof of Concept, Proof of Product, and Proof of Adoption. Each SoD stage comprises associated: activities, outputs, and outcomes common to many development projects. Successful outcomes for SoD stages would support that a product is (respectively): Feasible, Realizable, and Beneficial. Copyright National Institute on Disability, Independent Living, and Rehabilitation Research 9/16/2015 7

SoD Overview SoD Uses might include: proposal preparation; grant management, monitoring, reporting, and evaluation; portfolio assessment and management; programmatic analysis and planning; and dissemination. The proposed SoD are intended to provide: Descriptive Guidances rather than Prescriptive Requirements. Copyright National Institute on Disability, Independent Living, and Rehabilitation Research 9/16/2015 8

SoD in Detail The following slides provide information pertaining to SoD concepts along with examples. Each slide includes a descriptive title (upper right hand corner). Many slides also include graphics. When graphics and text occur on the same slide: please read left-hand text boxes from top-to-bottom; and then read right-hand text boxes from top-to- bottom. Copyright National Institute on Disability, Independent Living, and Rehabilitation Research 9/16/2015 9

SoD in Detail The following slides provide exemplars and examples to stimulate thought and encourage comment. Many details have been omitted to avoid undue complexity. We have included many (but not all) important concepts. We have tried to present these concepts in a reasonably logical order. We have also omitted many references used to construct the proposed SoD. Copyright National Institute on Disability, Independent Living, and Rehabilitation Research 9/16/

SoD Stages PROOF OF ADOPTION PROOF OF CONCEPT PROOF OF PRODUCT D First Stage activities transform the Product IDEA into a Proof of Concept. Second Stage activities transform the Proof of Concept into a Proof of Product The development Product Idea is determined by the investigator. For example, the Product Idea might derive from a literature review, prior investigator work, market research, or some combination. Development (D) comprises 3 Stages, 3 Outcomes, and associated Stage Outputs and Activities. PRODUCT IDEA Stage 1 Stage 2 Stage 3 Third Stage activities transform a Proof of Product into an Adopted Product. Copyright National Institute on Disability, Independent Living, and Rehabilitation Research 9/16/

SoD Outcomes PROOF OF ADOPTION PROOF OF CONCEPT PROOF OF PRODUCT A successful First Stage outcome would support that the Product Idea is Feasible. For example, key product concepts, components, & systems might be demonstrated. A successful Second Stage outcome would support that the Product is Realizable. For example, an integrated & working prototype might be demonstrated. A successful Third Stage outcome would support that a Product is Beneficial. For example, a Product might be demonstrated to be substantially adopted and used for its intended purpose by its target population. Stage 1 Stage 2 Stage 3 PRODUCT IDEA Copyright National Institute on Disability, Independent Living, and Rehabilitation Research 9/16/

Target Market(s) Products are associated with a Target Market(s) and Intended Use(s). The Target Market(s) comprises those persons who will use the Product. The Target Market might for example comprise Individuals with Disability (IWD) using the Product; or caregivers, providers, researchers, developers or others using (or purchasing) the Product in a manner benefitting IWD. Product Examples IWD (Outcomes) Researchers (OT, PT) Providers (Engineers, Designers) Developers Target Market Examples (Wayfinding) Device (Screen Reader) Application (Service Provision) Method (Wheelchair) Standard (Usability Assessment) Tool 9/16/2015 Copyright National Institute on Disability, Independent Living, and Rehabilitation Research 13

Product Types Development Products can usually be associated with “Types” Examples: Models (e.g. for service provision) Tools (e.g. for usability measurement) Standards (e.g. for manual wheelchairs) Applications (e.g. for task planning) Devices (e.g. for wayfinding) … Copyright National Institute on Disability, Independent Living, and Rehabilitation Research 9/16/

Partner Organizations Development Projects are generally accomplished in collaboration with Partner organizations. Examples: Universities Manufacturers Distributors Professional Organizations Standards Organizations … Copyright National Institute on Disability, Independent Living, and Rehabilitation Research 9/16/

Products and Partners Product Adoption (and earlier Stage activities & outcomes) is facilitated by collaboration with one or more Partner Organizations. In practice, the Partner Organization often provides the path/means by which a Product reaches its Target Market. Product Examples Play Store Manufacturer Online Distributor Professional Organization Standards Organization Partner Examples (Wayfinding) Device (Screen Reader) Application (Service Provision) Model (Wheelchair) Standard Copyright National Institute on Disability, Independent Living, and Rehabilitation Research 9/16/

Partner Initiated Requirements Partner organizations often introduce investigators to Product requirements that are relevant to one or more stage’ activities, outputs, and outcomes. For this (and closely-related) reason(s), early identification of and collaboration with appropriate Partner organization(s) often improves outcomes for all SoD stages. Partner Organization (Product) Examples for Design, Testing of AT Company Inc. (Wayfinding Device) for Distribution of Online Software Distributor (Task Application) for Use in Practice Professional Association ([Service Provision] Method) Stage Requirement Examples Protocols for Development ofStandard Organization ([Wheelchair] Standard) Copyright National Institute on Disability, Independent Living, and Rehabilitation Research 9/16/

Stage Assessments Each stage normally has investigator-generated requirements which they propose to achieve. Corresponding, each stage normally has investigator-generated assessments used to evaluate whether requirements have been met. PROOF OF ADOPTION PROOF OF CONCEPT PROOF OF PRODUCT Stage 1 Stage 2 Stage 3 PRODUCT IDEA 9/16/2015 Copyright National Institute on Disability, Independent Living, and Rehabilitation Research 18

Stage 1 Assessment(s) Meeting First Stage requirements supports that the Proof of Concept is Feasible Examples: design evaluation (assess device compliance to design requirements; e.g. of device) bench testing (assess compliance to design, performance,… specifications; e.g. devices, applications) standards testing (assess compliance to safety, design, performance,… standards; e.g. of devices, applications) psychometric testing (assess validity, reliability,…; e.g. of tools) user testing (assess usability, utility,…; e.g. of methods, devices, applications…) … PROOF OF ADOPTION PROOF OF CONCEPT PROOF OF PRODUCT Stage 1 PRODUCT IDEA Copyright National Institute on Disability, Independent Living, and Rehabilitation Research 9/16/

Stage 2 Assessments Meeting Second Stage requirements supports that the Product is Realizable. Examples: user trials (usability, utility,… assessed by users) provider trials (e.g. efficiency, effectiveness, consistency… assessed by providers) partner trials (e.g. product design requirements… assessed by Partner) … PROOF OF ADOPTION PROOF OF CONCEPT PROOF OF PRODUCT Stage 2 PRODUCT IDEA Copyright National Institute on Disability, Independent Living, and Rehabilitation Research 9/16/

Stage 3 Assessments Meeting Third Stage requirements supports that the Product is Beneficial. A member of the Target Population only Benefits if the product is Used (and Retained) for its Intended Purpose. Examples: Awareness (of product) Persuasion (positive disposition towards product) Decision (to obtain product) Use (to try product) Retention (to continue using product) Breadth (of target population adopting the product) … PROOF OF ADOPTION Stage 3 PROOF OF CONCEPT PROOF OF PRODUCT PRODUCT IDEA Copyright National Institute on Disability, Independent Living, and Rehabilitation Research 9/16/

Product Adoption and Adoption Metrics Specific adoption metrics depend on the Product and its Target Population and Intended Use. One or more specific adoption metrics can be identified by the investigator for each generic adoption metric. Awareness of… Persuasion towards… Decision to adopt… Use of… Retention of… E.g. Hits, citations… E.g. Disposition towards E.g. Purchase, downloads E.g. Use at one month E.g. Use at one year Generic Adoption Metrics Specific Metric Examples Copyright National Institute on Disability, Independent Living, and Rehabilitation Research 9/16/

Product Adoption and Technology Transfer Technology Transfer comprises the plans and strategies used to overcome barriers and to facilitate adoption of a development Product by its Target Market and for its Intended Use. Technology transfer generally involves collaboration with Partner organizations. Example: RERC investigators are required to prepare a Technology Transfer Plan by the end of the grant’s 1 st year. Example: SBIR investigators are required to prepare Commercialization [Technology Transfer] Plans in the beginning of Phase II (corresponding to the beginning of SoD Stage 2). The NIDILRR-funded Center on Knowledge Translation for Technology Transfer … Center on Knowledge Translation for Technology Transfer Provides guidance (based on best practices) to other NIDILRR-funded grantees to support development of Technology Transfer Plans. Copyright National Institute on Disability, Independent Living, and Rehabilitation Research 9/16/

Stage Iteration In-Stage Iteration (of some or all Stage activities) is normal and expected. Across-Stage Iteration may occur for good reason, but is not typical; and may (or may not) indicate that a development project has encountered a significant problem. Consequently, Across-Stage iteration would normally involve NIDILRR notification and might require NIDILRR approval. For example, First Stage design requirements (e.g. for a device) might be found to be inconsistent with a Partner initiated requirements (e.g. for performance) found in the Second Stage. The investigator might wish to modify First Stage requirements and repeat the stage. The investigator would start by discussing their plan with their NIIDILRR project officer. Copyright National Institute on Disability, Independent Living, and Rehabilitation Research 9/16/

Research & Development Research and Development are often inter- dependent and inter-twined. However, Products deriving from Research or Development are very different. PROOF OF ADOPTION PROOF OF CONCEPT PROOF OF PRODUCT Stage 2 PRODUCT IDEA Stage 1 Stage 3 Product Adoption Study Development Products (E.g. Adopted Models, Methods, Tools, Standards, Apps, Devices,…) Research Products (E.g. Publications, Proceedings, Seminars,…) Product User Trialing Study Market Research Study System Bench Testing Study New Knowledge 25

Development Example: SBIR Grants SBIR grants and RERC development projects provide a nice illustration of SoD concepts. SBIR grantees are normally small businesses. SBIR Phase I and Phase II correspond closely to SoD First and Second Stages. SBIR investigators are required to write a Commercialization [Technology Transfer] Plan in Phase II. The Plan is the means by which the SoD Third Stage (Product Adoption) is to be accomplished. ADOPTION PRODUCT IDEA PROOF OF CONCEPT PROOF OF PRODUCT Stage 1 Stage 2 Stage 3 Phase I < 1 Year Phase II 2 Years Phase II Commercialization Plan 9/16/2015 Copyright National Institute on Disability, Independent Living, and Rehabilitation Research 26

Development Example: RERC Grants RERC grantees are normally universities. Target Markets for RERC development projects may include (for example) IWD, caregivers, providers, researchers, or developers. Product Types for RERC development projects may include (for example) models, methods, tools, standards, applications, or devices. RERCs investigators are required to develop a Technology Transfer Plan in the first grant year. PROOF OF ADOPTION IDEA PROOF OF CONCEPT PROOF OF PRODUCT Stage 1 Stage 2 Stage 3 RERC Partner Organizations may include (for example) manufacturers, distributors, professional organizations, or standards organizations. RERC development projects may encompass one ore more of the SoD Stages (Proof of Concept, Proof of Product, and Proof of Adoption). Copyright National Institute on Disability, Independent Living, and Rehabilitation Research 9/16/

Thank you! We appreciate your assistance! Copyright National Institute on Disability, Independent Living, and Rehabilitation Research 9/16/