1. Somani Patnaik 1, Emma Brunskill 1, William Thies 2 1 Massachusetts Institute of Technology 2 Microsoft Research India Accuracy of Data Collection on Mobile Phones: A Study of Forms, SMS, and Voice

2. Mobile Data Collection Broad applications in – – – – Mobile banking Microfinance Healthcare Environmental monitoring Benefits  Immediate digitization  Fast and cheap  Environment friendly  Flexible questionnaires

3. Study of data collection interfaces Comparison of three interfaces for health data collection

4. 1. Electronic Forms Interface

5. General Strengths  Easy patient identification  Ongoing cost is low (SMS or data plan)  Can store visits when connectivity is poor General Weaknesses  Requires programmable phones  Requires basic literacy skills  Hard to alter survey questions  Hard to enter in free-form notes  Application can be deleted by user Consist of numeric fields and multiple-choice menus. Can be implemented in Java or a native phone platform.

6. 2. SMS Interface

7. Sending a structured SMS messages to a server Logical fields separated by delimiters in the message General Strengths  Can be used with any phone  Ongoing cost is low (SMS or data plan)  Many workers familiar with SMS General Weaknesses  Requires basic literacy skills  Changing survey requires new cue card  Quite easy to fake visits (copy old SMS)  Hard to enter in free-form notes

8. 3. Live Operator Interface

9. A normal telephone call Live human operator that enters the data into a centralized spreadsheet General Strengths  No literacy required of workers  Can be used with any phone  Hard to fake a visit: operator can ask new questions General Weaknesses  Ongoing cost of operator salary  Voice plans often higher cost than SMS  Awkward 3-way social interaction

10. Operations  Setup time - Electronic forms require application, which requires either an Internet-enabled phone or an external computer  Training time – Voice interface requires least amount of education and background  System coverage and reliability - voice most reliable but require sufficient number of operators.  Flexibility - Ability to modify the data collection interface, fix an error, improve usability etc

11. Effectiveness Characterized by two factors-  Data should not be intentionally faked by the user  The data should be accurate(not intentionally faked) Unfortunately, quite easy to fake data in SMS systems, just requires copying and pasting prior messages Faking data on electronic systems slightly harder and most difficult in voice. Voice also allows correcting previous visits

12. Cost Comparison Electronic phone requires a programmable phone(such as a Java enable phone or windows phone) Voice has the ongoing cost of the operator Cost of live-operator in India proves to be cost-effective Decreased cost of voice-only handsets, training time and literacy requirements of health workers compensate the cost

13. Cost Analysis

14. ..Cost Analysis

15. Context: Rural Tuberculosis Treatment With local partners, working to improve tuberculosis treatment in rural Bihar. Monitoring the patients symptoms remotely by collecting data remotely.

16. Study Participants 13 health workers and hospital staff (Gujarat, India) Age (Median) EducationCell Phone Experience Training  Health workers: big groups, 6-8 hours  Hospital staff: small groups, 1-2 hours  Every user made two error-free reports on each interface

17. Testing and Results Testing  Tested in pairs, one patient on data entry, and the other being the fake patient  Two complete patient–worker interactions for electronic forms and SMS interfaces Results

18. Error rate higher in health workers as compared to hospital staff Two possible reasons:  Hospital staff more educated and old  Difference in training

19. Sources of Error Types of errors Misplacement of decimal point in the temperature entry Non-revision of cue cards for the SMS interface Putting wrong patient identity when using SMS Usability barriers small keys scrolling/selection SMS encoding

20. Conclusions Accuracy of mobile data collection demands attention  5% error rates for those lacking experience There exist cases where a live operator makes sense  Can be cost effective, esp. for short calls Study has limitations  Small sample size  Varied education, phone experience, training of participants Future work  Distinguish factors responsible for error rates  Compare to paper forms, Interactive Voice Response