Welcome! If you joined the conference via telephone, please enter your audio pin if you havent already. (This is an example, your audio pin will be different) If you joined with a microphone and headset or speakers (VoIP), please be sure your device volume settings are properly adjusted. If your mic & headset is not working, please try unplugging and re-plugging in your device. If you have any questions, please enter them in the questions module. We will begin promptly at the top of the hour.

DOC110: Evaluating Document Assembly Projects September 26, 2008

GoToWebinar Interface Maximize/minimize your screen with the chevron symbol Telephone participants need to enter their audio pin Please share your questions and knowledge! Ask a question in the questions log Raise your hand

Panelists Kate Bladow, Pro Bono Net Steve Rapp, Idaho Legal Aid Services Bristow Hardin, Legal Services Corp.

Todays Training What is evaluation? Why evaluate? How do you evaluate a program? Planning for an evaluation The process Case study: Idaho Legal Aid Services What is LSC looking for in evaluation plans and reports?

Evaluation Defined Systematically collecting data to answer questions of interest.

Reasons to Evaluate a Program To find out if it was successful To build buy-in To document lessons learned and potential best practices To identify areas for improvement To justify the cost and effort

Evaluation Process 1. Draft an evaluation plan. 2. Design and test data collection instruments. 3. Collect baseline data. 4. Collect additional data. 5. Analyze data and report findings. 6. Integrate necessary changes. 7. Repeat steps.

Draft an Evaluation Plan

Example Evaluation Plan Project Goal: Use A2J author and HotDocs to create guided interviews and court form templates that improve the ability of low income people to easily, effectively and efficiently produce and file court forms without the assistance of an attorney. Project ObjectivesStrategies/ActivitiesEvaluation QuestionsEvaluation Data Objective 1 Use A2J Author and HotDocs to create guided interviews and court form templates that enable pro se litigants to easily produce accurate pro se pleadings. Produce [insert number] guided interviews and [insert number] interactive court forms. Identify forms/interviews to automate Hire forms developer to help develop guided interviews and automated forms Work with courts to ensure that interviews/forms are accurate, consistent with court requirements and will be acceptable to the courts. Conduct usability testing of templates to ensure users (clients and/or advocates) can easily understand and use the interviews/forms. (Note: forms should be at the eight-grade reading level of lower.) Modify templates to address any problems identified in usability testing Publish forms/interviews on statewide website What forms/interviews were selected for automation? Was developer hired to develop guided interviews and automated forms? What interviews were produced? What forms were automated? Have courts confirmed that the forms are accurate, consistent with court requirements and acceptable for use? What usability testing was conducted? What were the results of the usability testing? What changes were made to the automated forms based on the initial testing? What automated forms and guided interviews been published on the statewide website? To what extent have the automated forms and guided interviews made it easier for self- represented litigants to prepare accurate court pleadings? Confirmation that developer hired. List of interviews and automated court forms produced. Court confirmation that forms are forms are accurate, consistent with court requirements and acceptable for use Description of usability testing Results of usability testing Description of notable changes made to forms and interviews following initial testing List of forms/interviews published on statewide website User interviews/surveys with: Users Court staff User feedback from the website Field observation data Objective 2 Increase the availability and use by low-income persons of automated forms Partner with courts and other stakeholders to develop and implement outreach initiatives to: foster courts acceptance of the forms maximize clients use of forms Partner with public libraries, courthouses and other groups to increase access points where clients can: access and complete forms receive assistance in completing forms What automated forms and guided interviews have been posted the statewide website? Was an outreach plan developed? What outreach activities were conducted? To what extent are the different forms being used by clients? How many judicial districts/courts have accepted the automated forms? How many new access points have been established where clients can access the forms? At how many access points can clients receive assistance in completing the forms? Copy of outreach plan Description of outreach activities Statistics from national server on number of times different forms have been accessed and/or downloaded List of judicial districts that have accepted the automated forms List of new access points (with description of services available to clients)

Design and test instruments Time Trials Focus Groups Usage Statistics Usability Tests Document Comparisons Interviews Surveys

Collect baseline data Allows you to compare before and after. Can be collected while the document assembly content is being developed. Keep this data collection consistent with your later data collection.

Collect additional data Happens after implementation. Stay consistent with your early data collection.

Analyze data and report findings Project Goals and Objectives Evaluation Data and Methodologies Summary of Major Accomplishments Recommendations and Future Steps In-Depth Analysis of Accomplishments Factors affecting project accomplishments Strategies to address major challenges Major lessons and recommendations

Integrate necessary changes Use the feedback that you gathered. Tweak the program and then collect more evaluation data to find out if it really was a positive change.

Repeat Plan Design CollectAnalyze Report Integrate

Case Study Idaho Legal Aid Services

Idaho Project Highlights Partnership between Idaho Legal Aid Services and the Idaho Supreme Court Funding provided by the Legal Services Corporation and the State Justice Institute Successful – good feedback and good filings ©2008 National Center for State Courts

A Conversation with LSC

Questions? Kate Bladow – Steve Rapp - Bristow Hardin –

Resources

DA and Legal Aid Specific Document Assembly Support LSC Evaluation Plan and Report Forms SelfHelpSupport.org

Overview and Theory W.K.K. Foundation Evaluation Handbook Outcome Measurement Resource Network Basic Guide to Program Evaluation Intelligent Measurement Blog Point K

Surveys LSNTAP Survey Tool Creating Effective Online Surveys Likert Scales

Usability Testing Usability Methods Assess Your Sites Navigation & Structure Useit.com

Focus Groups How to Use Focus Groups

Data Analysis Excel Skills for Advocates W.K.K Foundation Data Analysis How to Analyze Data Overview Introduction to Statistics The Complete Idiot's Guide to Statistics or

Attributions Image – Slide 6 – SMercury Image – Slide 8 – markymarko Image – Slide 10 – roboppy Image – Slide 10 – gergenzola Image – Slide 10 – Eric Spiegelman Image – Slide 10 – jimmedia Image – Slide 10 – cambodia4kids Image – Slide 10 – ifindkarma Images – Slide 13 – johnbullas Image – Slide 27 – Faeryan Content – Slides 7, 8, 10-13, – Adapted from Court Leadership and Self-Represented Litigation Package: Module 7