1. Welcome! If you joined the conference via telephone, please enter your audio pin if you havent already. (This is an example, your audio pin will be different) If you joined with a microphone and headset or speakers (VoIP), please be sure your device volume settings are properly adjusted. If your mic & headset is not working, please try unplugging and re-plugging in your device. If you have any questions, please enter them in the questions module. We will begin promptly at the top of the hour.

2. DOC110: Evaluating Document Assembly Projects September 26, 2008

3. GoToWebinar Interface Maximize/minimize your screen with the chevron symbol Telephone participants need to enter their audio pin Please share your questions and knowledge! Ask a question in the questions log Raise your hand

4. Panelists Kate Bladow, Pro Bono Net Steve Rapp, Idaho Legal Aid Services Bristow Hardin, Legal Services Corp.

5. Todays Training What is evaluation? Why evaluate? How do you evaluate a program? Planning for an evaluation The process Case study: Idaho Legal Aid Services What is LSC looking for in evaluation plans and reports?

6. Evaluation Defined Systematically collecting data to answer questions of interest.

7. Reasons to Evaluate a Program To find out if it was successful To build buy-in To document lessons learned and potential best practices To identify areas for improvement To justify the cost and effort

8. Evaluation Process 1. Draft an evaluation plan. 2. Design and test data collection instruments. 3. Collect baseline data. 4. Collect additional data. 5. Analyze data and report findings. 6. Integrate necessary changes. 7. Repeat steps.

9. Draft an Evaluation Plan

10. Example Evaluation Plan Project Goal: Use A2J author and HotDocs to create guided interviews and court form templates that improve the ability of low income people to easily, effectively and efficiently produce and file court forms without the assistance of an attorney. Project ObjectivesStrategies/ActivitiesEvaluation QuestionsEvaluation Data Objective 1 Use A2J Author and HotDocs to create guided interviews and court form templates that enable pro se litigants to easily produce accurate pro se pleadings. Produce [insert number] guided interviews and [insert number] interactive court forms. Identify forms/interviews to automate Hire forms developer to help develop guided interviews and automated forms Work with courts to ensure that interviews/forms are accurate, consistent with court requirements and will be acceptable to the courts. Conduct usability testing of templates to ensure users (clients and/or advocates) can easily understand and use the interviews/forms. (Note: forms should be at the eight-grade reading level of lower.) Modify templates to address any problems identified in usability testing Publish forms/interviews on statewide website What forms/interviews were selected for automation? Was developer hired to develop guided interviews and automated forms? What interviews were produced? What forms were automated? Have courts confirmed that the forms are accurate, consistent with court requirements and acceptable for use? What usability testing was conducted? What were the results of the usability testing? What changes were made to the automated forms based on the initial testing? What automated forms and guided interviews been published on the statewide website? To what extent have the automated forms and guided interviews made it easier for self- represented litigants to prepare accurate court pleadings? Confirmation that developer hired. List of interviews and automated court forms produced. Court confirmation that forms are forms are accurate, consistent with court requirements and acceptable for use Description of usability testing Results of usability testing Description of notable changes made to forms and interviews following initial testing List of forms/interviews published on statewide website User interviews/surveys with: Users Court staff User feedback from the website Field observation data Objective 2 Increase the availability and use by low-income persons of automated forms Partner with courts and other stakeholders to develop and implement outreach initiatives to: foster courts acceptance of the forms maximize clients use of forms Partner with public libraries, courthouses and other groups to increase access points where clients can: access and complete forms receive assistance in completing forms What automated forms and guided interviews have been posted the statewide website? Was an outreach plan developed? What outreach activities were conducted? To what extent are the different forms being used by clients? How many judicial districts/courts have accepted the automated forms? How many new access points have been established where clients can access the forms? At how many access points can clients receive assistance in completing the forms? Copy of outreach plan Description of outreach activities Statistics from national server on number of times different forms have been accessed and/or downloaded List of judicial districts that have accepted the automated forms List of new access points (with description of services available to clients)

11. Design and test instruments Time Trials Focus Groups Usage Statistics Usability Tests Document Comparisons Interviews Surveys

12. Collect baseline data Allows you to compare before and after. Can be collected while the document assembly content is being developed. Keep this data collection consistent with your later data collection.

13. Collect additional data Happens after implementation. Stay consistent with your early data collection.

14. Analyze data and report findings Project Goals and Objectives Evaluation Data and Methodologies Summary of Major Accomplishments Recommendations and Future Steps In-Depth Analysis of Accomplishments Factors affecting project accomplishments Strategies to address major challenges Major lessons and recommendations

15. Integrate necessary changes Use the feedback that you gathered. Tweak the program and then collect more evaluation data to find out if it really was a positive change.

16. Repeat Plan Design CollectAnalyze Report Integrate

17. Case Study Idaho Legal Aid Services

18. Idaho Project Highlights Partnership between Idaho Legal Aid Services and the Idaho Supreme Court Funding provided by the Legal Services Corporation and the State Justice Institute Successful – good feedback and good filings ©2008 National Center for State Courts

24. A Conversation with LSC

25. Questions? Kate Bladow – kbladow@probono.net Steve Rapp - steverapp@idaholegalaid.org Bristow Hardin – hardinb@lsc.gov

26. Resources

27. DA and Legal Aid Specific Document Assembly Support http://www.probono.net/dasupport http://www.probono.net/dasupport LSC Evaluation Plan and Report Forms http://tig.lsc.gov/eval_nonwebsite_grants.php http://tig.lsc.gov/eval_nonwebsite_grants.php SelfHelpSupport.org http://bit.ly/y4c4s http://bit.ly/y4c4s

28. Overview and Theory W.K.K. Foundation Evaluation Handbook http://bit.ly/1fg3n4 http://bit.ly/1fg3n4 Outcome Measurement Resource Network http://bit.ly/4rD2eY http://bit.ly/4rD2eY Basic Guide to Program Evaluation http://bit.ly/eHHjW http://bit.ly/eHHjW Intelligent Measurement Blog http://intelligentmeasurement.wordpress.com/ http://intelligentmeasurement.wordpress.com/ Point K http://bit.ly/HQL2k http://bit.ly/HQL2k

29. Surveys LSNTAP Survey Tool http://bit.ly/2ARmCF http://bit.ly/2ARmCF Creating Effective Online Surveys http://bit.ly/1bLpwg http://bit.ly/1bLpwg Likert Scales http://bit.ly/4nlZlh http://bit.ly/4nlZlh

30. Usability Testing Usability Methods http://usability.gov/methods/ http://usability.gov/methods/ Assess Your Sites Navigation & Structure http://bit.ly/1Bc1Pv http://bit.ly/1Bc1Pv Useit.com http://www.useit.com/ http://www.useit.com/

31. Focus Groups How to Use Focus Groups http://bit.ly/NHLBA http://bit.ly/NHLBA

32. Data Analysis Excel Skills for Advocates http://www.lsntap.org/OFF105 http://www.lsntap.org/OFF105 W.K.K Foundation Data Analysis http://bit.ly/44nU2h http://bit.ly/44nU2h How to Analyze Data http://bit.ly/3sRnvP http://bit.ly/3sRnvP Overview Introduction to Statistics http://bit.ly/3TpaMd http://bit.ly/3TpaMd The Complete Idiot's Guide to Statistics http://bit.ly/2t0Hen or http://bit.ly/4hRQok http://bit.ly/2t0Henhttp://bit.ly/4hRQok

33. Attributions Image – Slide 6 – SMercury98 http://www.flickr.com/photos/smercury98/ http://www.flickr.com/photos/smercury98/ Image – Slide 8 – markymarko http://www.flickr.com/photos/cybercafe/ http://www.flickr.com/photos/cybercafe/ Image – Slide 10 – roboppy http://www.flickr.com/photos/roboppy/ http://www.flickr.com/photos/roboppy/ Image – Slide 10 – gergenzola http://www.flickr.com/photos/gergenzola/ http://www.flickr.com/photos/gergenzola/ Image – Slide 10 – Eric Spiegelman http://www.flickr.com/photos/ericspiegelman/ http://www.flickr.com/photos/ericspiegelman/ Image – Slide 10 – jimmedia http://www.flickr.com/photos/jimmediaart/ http://www.flickr.com/photos/jimmediaart/ Image – Slide 10 – cambodia4kids http://www.flickr.com/photos/cambodia4kidsorg/ http://www.flickr.com/photos/cambodia4kidsorg/ Image – Slide 10 – ifindkarma http://www.flickr.com/photos/ifindkarma/ http://www.flickr.com/photos/ifindkarma/ Images – Slide 13 – johnbullas http://www.flickr.com/photos/johnbullas/ Image – Slide 27 – Faeryan http://www.flickr.com/photos/faeryan/ http://www.flickr.com/photos/faeryan/ Content – Slides 7, 8, 10-13, 16-18 – Adapted from Court Leadership and Self-Represented Litigation Package: Module 7 http://bit.ly/3qjLJ4 http://bit.ly/3qjLJ4