NCHRP (47) - MTAP Survey Tool Used to Assess FTA Contractor Performance of State DOT Triennial and Other FTA Reviews - An Update Biennial FTA State Programs Meeting / State Public Transit Partnerships Conference August 7, 2103 KFH Group, Inc. Beth Hamby Sue Knapp

Background To assist FTA with improving State Review processes, MTAP collects input from State DOTs on their experience following each FTA review via an online survey tool. Survey was created in Vovici (used by AASHTO for other surveys). Value of survey has been somewhat limited by: ▫open-ended answer formats ▫challenges in using Vovici

Research Objective: Update MTAP’s online survey to: ▫improve efficiency / ease of completion ▫increase the value added nature of the process (improve results for ease of use, benefits) ▫incorporate technology improvements ▫provide for more detailed responses to 2-3 specific questions of interest to MTAP, SCOPT, FTA - not just a "comments" field

Project Tasks Task 1 – Assess Survey Tool Improvement Needs Task 2 – Revise Survey Content & Structure Task 3 – Assess Available Survey Technology Options Task 5 – Set Up & Test Revised Survey Using Selected Technology Task 6 – Prepare Final Report

Project Status Task 1 – Assess Survey Tool Improvement Needs – completed ▫Reviewed current survey tool ▫Reviewed reports generated from current tool ▫Gathered feedback/suggestions from  States/MTAP Committee  FTA Office of Oversight and Program Guidance  MTAP Coordinator

Project Status Task 2 – Revise Survey Content & Structure – nearing completion ▫Modified existing questions – standardized responses where possible to make analysis easier ▫Added new questions ▫Prepared revised survey for review by MTAP/ Steering Committee; sent late July

Draft Revised Survey Numerous new questions seeking specific feedback ▫Basic experience of reviewer ▫Consultant performance on specific compliance topics ▫For each scaled rating (Excellent/ Good/ Fair/ Poor), asks why (open-ended) ▫Extent of Regional participation ▫Effectiveness of entrance/exit meetings ▫SMR workshop effectiveness ▫Interest in follow-up discussion with MTAP or FTA Standardized responses; limited open-ended questions to explanation and “other” Reworded some questions to align with responses

Draft Revised Survey 1.Contact and Background Info ▫Added: Basic experience of respondent 2.Consultant Performance ▫Added: Fields for explanation 3.General Experience with Review ▫Added: Regional office participation ▫Added: Entrance/Exit meetings ▫Added: Overall organization and clarity 4.SMR-Specific Feedback ▫Added: Respondent’s participation ▫Workshop/workbook effectiveness ▫Additional training/tech assistance needs 5.Additional Feedback

Project Status Task 3 – Assess Available Survey Technology Options – under way ▫Criteria include:  User-friendliness to create & administer survey  User-friendliness to complete & submit surveys  Functionality/customization options  Reporting capabilities  Cost

Survey Approaches General options for getting feedback from states: ▫Online survey tools – basic to advanced  Focus of our assessment ▫Pen & paper  Too labor intensive ▫Telephone interviews  May be appropriate for follow-up

Basic Online Survey Tools Examples include: ▫SurveyMonkey Basic / Select / Gold ▫SurveyMoz Free / Premium ▫SurveyGizmo Basic ▫QuestionPro Professional

Basic Online Survey Tools Advantages: ▫Easy to learn and use ▫Provides some skip logic functionality ▫Provides some basic cross-tab analysis ▫Can export results to Excel for additional analysis ▫Inexpensive ($0 - $25/month)

Basic Online Survey Tools Disadvantages: ▫Limited collaboration capabilities ▫Limited survey customization options ▫Limited reporting / analysis functions – need to export to third party application Basic tools are becoming increasingly sophisticated (e.g., SurveyMoz) Some offer higher levels of customization for a higher cost (e.g., SurveyMonkey Platinum - $65/month)

Advanced Online Survey Tools What MTAP is currently using (Vovici) Known as Enterprise Feedback Management (EFM) Systems Examples include: ▫Key Survey ▫Qualtrics ▫QuestionPro Corporate ▫SurveyGizmo Enterprise ▫Vovici

Advanced Online Survey Tools Advantages: ▫Allows for greater collaboration in survey administration (behind the scenes)  Multiple user accounts possible  Can share surveys with other survey administrators  Private results through secure dashboard accounts ▫Greater flexibility in branding and customizing ▫Can build sophisticated logic into survey ▫Allows for complex analysis ▫Custom reports can be developed within the tool

Advanced Online Survey Tools Disadvantages: ▫Complex interface ▫Requires training / steeper learning curve ▫May require frequent use to maintain skills ▫May require experience in Cascading Style Sheets (CSS) to customize look and feel of survey ▫Expensive ($100 or more/month – few disclose pricing online)  However, no out-of-pocket cost to MTAP for Vovici; AASHTO already pays for it for other uses

Next Steps Fine-tune survey questions with input from MTAP/ Committee Finish technology assessment Set up & test revised survey using selected technology Prepare final report

For more information or feedback Beth Hamby KFH Group